๐ Introduction
Star Defender is more than just a space shooter game—it's a living laboratory where artificial intelligence learns to play in real-time. This interactive demonstration showcases one of the most fascinating areas of machine learning, which is reinforcement learning, where an AI agent learns optimal strategies through experience, trial, and error. You can find the full code of Star Defender at the end of this article.
Unlike traditional games where the computer follows pre-programmed rules, Star Defender features an AI that genuinely learns and improves. You can watch as it transforms from a complete novice making random moves into a skilled player that dodges enemies, times its shots, and maximizes its score—all without being explicitly programmed on how to play.
This application serves a dual purpose. First, it functions as an engaging arcade game that humans can play using keyboard controls. Second, it operates as an AI training ground where a reinforcement learning agent learns to master the game through repeated attempts and continuous learning.
By comparing human gameplay with AI learning patterns, we gain unique insights into how machines develop intelligent behavior and how their learning process differs from human intuition. The application provides a transparent window into the AI's decision-making process, allowing viewers to observe the gradual emergence of strategic behavior from initially random actions.
๐ฎ Part 1: The Game - Star Defender
Game Overview
Star Defender is a classic arcade-style space shooter with a modern, neon-aesthetic twist. The player controls a cyan triangular spaceship positioned at the bottom of the screen, facing endless waves of enemy spacecraft descending from above. The core objective is simple yet challenging: survive as long as possible while destroying enemies to accumulate points. The game combines fast-paced action with strategic positioning, requiring players to balance offensive aggression with defensive caution.
Core Game Mechanics
Player Ship:
The player's spacecraft is represented as a glowing cyan triangular vessel that serves as the avatar for both human and AI players. The ship leaves a fading trail effect as it moves across the screen, creating a visual history of its recent movements. The spacecraft can move horizontally across the screen using either arrow keys for human players or AI-controlled decisions for the learning agent. The ship fires yellow bullets upward toward incoming enemies and begins each game session with three lives, providing multiple chances to improve performance.
Enemies:
The game features three distinct enemy types, each with unique characteristics that create varied tactical challenges. The first enemy type is the red enemy, which is small and fast. This enemy measures thirty by thirty pixels and moves at a speed of two units per frame. When destroyed, it awards ten points to the player. Its behavior is characterized by quick descent, making it harder to hit with bullets.
The second enemy type is the orange enemy, which is large and slow. This enemy measures forty by forty pixels and moves at a speed of one point five units per frame. When destroyed, it awards twenty points to the player. Its behavior makes it an easier target due to its size and slower movement, but it offers a higher reward to compensate.
The third enemy type is the magenta enemy, which is medium-sized and balanced. This enemy measures twenty-five by twenty-five pixels and moves at a speed of three units per frame. When destroyed, it awards fifteen points to the player. Its behavior is characterized by fast movement despite its smaller size, creating a balanced challenge.
All enemies exhibit a "wobble" behavior where they oscillate horizontally as they descend down the screen. This wobbling motion makes them more challenging to hit with bullets and creates dynamic gameplay patterns that require players to anticipate enemy positions rather than simply aiming at their current location.
Weapons System:
The player's weapon system fires yellow projectile bullets that travel upward toward enemies. The fire rate is limited by a cooldown mechanism that requires ten frames to pass between consecutive shots, preventing players from creating an impenetrable wall of bullets. Each bullet travels upward at ten units per frame, moving quickly enough to intercept descending enemies. When a bullet makes contact with an enemy, it causes instant destruction of that enemy. Each successful hit triggers particle explosion effects that provide satisfying visual feedback for the player's accuracy.
Power-Ups:
Power-ups appear as green rotating boxes that spawn periodically throughout the game. When collected by the player, these power-ups grant one additional life, extending the player's survival potential. Power-ups spawn randomly every four hundred frames, which translates to approximately every six to seven seconds of gameplay. These collectibles add strategic depth to the game by creating risk-versus-reward decisions where players must decide whether to maintain a safe position or move into potentially dangerous areas to collect valuable extra lives.
Lives & Game Over:
Each game session begins with three lives that represent the player's health pool. The player loses one life for each collision with an enemy spacecraft. The game ends when all lives have been depleted, forcing the player to restart if they wish to continue. The application tracks the high score across multiple game sessions, providing a persistent goal for players to surpass their previous best performances.
Visual Design & Effects
Star Defender features a retro-futuristic aesthetic that combines classic arcade nostalgia with modern visual effects, creating an engaging and visually appealing experience.
Background:
The background consists of a deep space gradient that transitions from dark blue to purple, creating a sense of depth and cosmic atmosphere. The background contains one hundred animated stars that scroll downward at varying speeds, simulating the sensation of traveling through space. This parallax scrolling effect creates a sense of motion through space even when the player's ship remains stationary. The stars feature different sizes and opacity levels to create an illusion of depth, with brighter and larger stars appearing closer while dimmer and smaller stars seem more distant.
Particle System:
Every significant action in the game triggers particle explosions that provide visual feedback and enhance the game's overall polish. When the player shoots, small yellow particles burst from the ship's position, creating a muzzle flash effect. When an enemy is destroyed, thirty particles in that enemy's color explode outward from the point of destruction, creating a satisfying explosion effect. When collisions occur between the player and enemies, forty white particles create dramatic impact effects that clearly communicate the damage event. When power-ups are collected, twenty green particles celebrate the pickup, providing positive reinforcement for the player's successful collection.
These particles have realistic physics simulation applied to them. Each particle has an initial velocity in a random direction, creating a natural explosion pattern. The particles experience gradual deceleration, with each particle retaining ninety-eight percent of its velocity per frame, causing them to slow down over time. The particles also feature fading opacity over time, gradually becoming transparent before disappearing. These physics properties combine to create natural-looking dispersal patterns that enhance the game's visual appeal.
Glow Effects:
All ships and bullets have neon glow effects created using shadow blur rendering techniques. These glows are color-coded to make different game elements easily distinguishable: cyan represents the player's ship, yellow represents bullets, and red, orange, or magenta represent the different enemy types. These glow effects create vibrant, eye-catching visuals that evoke classic arcade aesthetics while maintaining modern visual standards. The glowing effects enhance readability by making game elements stand out clearly against the dark background, and they contribute to the overall game feel by making actions feel more impactful and energetic.
UI Elements:
The user interface features neon-bordered panels with semi-transparent backgrounds that maintain the game's aesthetic while providing clear information. The interface displays real-time score, high score, and remaining lives in prominent positions. A mode indicator shows the current play state, displaying whether the game is idle, being played by a human, or being controlled by the AI agent. The entire interface uses a cyan and magenta color scheme throughout, maintaining visual consistency with the game's retro-futuristic aesthetic.
๐ง Part 2: Reinforcement Learning - The Theory
What is Reinforcement Learning?
Reinforcement Learning is a machine learning paradigm that is fundamentally different from traditional supervised learning approaches. Instead of learning from labeled examples where correct answers are provided, an RL agent learns by interacting with an environment and receiving feedback in the form of rewards or penalties. This learning approach more closely mimics how humans and animals learn through experience.
Think of it like training a dog to perform tricks. You don't show the dog exactly how to sit by providing a detailed instruction manual. Instead, you reward the dog when it sits correctly, perhaps with a treat or praise. Over time, through repeated trials, the dog learns that sitting leads to positive outcomes in the form of treats. Eventually, the dog sits on command without needing immediate rewards because it has internalized the behavior.
Similarly, our AI agent doesn't receive explicit instructions on how to play Star Defender. It doesn't have access to a strategy guide or pre-programmed rules for optimal play. Instead, it discovers effective strategies through experience by trying different actions, observing the consequences, and gradually learning which behaviors lead to success.
The RL Framework: Key Components
Every reinforcement learning system consists of fundamental elements that work together to enable learning. Understanding these components is essential to comprehending how the AI agent learns to play Star Defender.
The Agent
The agent is the decision-maker in the reinforcement learning system, which in our case is the AI player controlling the spaceship. The agent observes the current state of the environment, which includes information about player position, enemy locations, and other relevant game data. Based on these observations, the agent chooses actions according to its current policy, which is its strategy for selecting actions in different situations. Over time, the agent learns from its experiences, gradually improving its policy to achieve better performance.
The Environment
The environment is the game world that the agent interacts with, which in Star Defender includes the game canvas, enemies, bullets, power-ups, and all game mechanics. The environment responds to the agent's actions by updating the game state accordingly, such as moving the player's ship when the agent chooses a movement action. The environment provides state information to the agent after each action, allowing the agent to perceive the consequences of its decisions. The environment also generates rewards based on the agent's performance, providing the feedback signal that drives learning.
State
The state is a representation of the current situation in the game that captures all relevant information needed for decision-making. In Star Defender, the state includes the player's position discretized into grid cells, the location of the closest enemy, and the relative position between the player and threats. The state can be continuous, with infinite possible values, or discrete, with a finite set of possible states. The state representation must capture all relevant information for decision-making while remaining computationally manageable, as overly complex state spaces can make learning impossibly slow.
Action
Actions are the choices available to the agent at each decision point. In Star Defender, the available actions include moving left, moving right, shooting a bullet, or staying in the current position. These actions define the agent's capability to influence the environment and determine what strategies are possible. The action space can be discrete, with a finite set of options like in our game, or continuous, with infinite possible actions like controlling the exact speed of movement.
Reward
The reward is a feedback signal indicating the quality of the agent's action in the current situation. Positive rewards encourage specific behaviors by signaling that an action led to a desirable outcome. Negative rewards, also called penalties, discourage behaviors by signaling that an action led to an undesirable outcome. The reward function shapes the agent's learning direction by defining what "good" and "bad" mean in the context of the task. Careful reward design is critical because the agent will optimize for whatever reward signal it receives, even if that doesn't align with the true objective.
Policy
The policy is the strategy that maps states to actions, defining how the agent behaves in different situations. The policy can be deterministic, always choosing the same action in a given state, or probabilistic, choosing actions according to a probability distribution. The policy represents the agent's learned knowledge about how to play the game effectively. The ultimate goal of reinforcement learning is to find the optimal policy that maximizes the expected cumulative reward over time.
The Reinforcement Learning Loop
The learning process follows a continuous cycle that repeats throughout training. First, the agent observes the current state by perceiving the current game situation, including player position and enemy locations. Second, the agent chooses an action based on its current policy, balancing exploration of new strategies with exploitation of known good strategies. Third, the agent executes the chosen action in the environment, such as moving, shooting, or staying still. Fourth, the agent receives a reward that provides feedback on the action's quality, with positive values for good outcomes and negative values for bad outcomes. Fifth, the agent updates its knowledge by adjusting learned values based on the experience, improving its policy for future decisions. After this update, the cycle returns to the first step and continues indefinitely.
This cycle repeats thousands of times during training, with the agent gradually improving its performance through accumulated experience. Each iteration provides new information that refines the agent's understanding of which actions work well in which situations.
Q-Learning: The Algorithm Behind Our AI
Our AI agent uses Q-Learning, which is one of the most fundamental and powerful reinforcement learning algorithms. The "Q" in Q-Learning stands for "quality," representing how good a particular action is when taken in a particular state.
The Q-Function:
The Q-function, denoted as Q(s, a), represents the expected total reward when taking action 'a' in state 's' and following the optimal policy thereafter. This function essentially answers the question: "If I take this action in this situation, how much total reward can I expect to accumulate?"
The Q-Learning Update Rule:
The Q-learning algorithm updates its Q-values using the formula: Q(s, a) ← Q(s, a) + ฮฑ [r + ฮณ · max Q(s', a') - Q(s, a)]. This formula may look complex, but each component serves a specific purpose in the learning process.
The term Q(s, a) represents the current quality estimate for the state-action pair being updated. This is the agent's current belief about how good this action is in this state.
The term ฮฑ (alpha) is the learning rate, set to zero point one in our implementation. This parameter controls how much new information overrides old knowledge. If the learning rate is too high, learning becomes unstable because new experiences cause dramatic changes in Q-values. If the learning rate is too low, learning becomes extremely slow because each experience has minimal impact. The value of zero point one provides a good balance between stability and learning speed.
The term r represents the immediate reward received from taking the action. This is the direct feedback from the current action, such as points earned for destroying an enemy or penalties for getting too close to threats.
The term ฮณ (gamma) is the discount factor, set to zero point ninety-five in our implementation. This parameter determines how much the agent values future rewards compared to immediate rewards. A discount factor of zero means the agent only cares about immediate rewards and ignores future consequences. A discount factor of one means future rewards are equally important as immediate rewards. Our value of zero point ninety-five means the agent highly values future rewards, encouraging it to consider long-term consequences of its actions.
The term max Q(s', a') represents the best possible future value, which is the maximum Q-value achievable from the next state. This represents the best outcome the agent can achieve from its new position after taking the current action.
What This Means:
The Q-learning formula combines immediate reward with estimated future value, gradually building a comprehensive understanding of which actions lead to long-term success. The algorithm doesn't just focus on immediate gratification; instead, the agent learns to value actions that set up future opportunities. For example, the agent might learn that moving away from an enemy provides a small immediate reward for safety, but more importantly, it positions the agent to survive longer and score more points in the future.
Exploration vs. Exploitation Dilemma
One of the most fascinating challenges in reinforcement learning is balancing two competing objectives: exploration and exploitation.
Exploration
Exploration involves trying new and potentially better strategies that the agent hasn't fully evaluated yet. This means discovering unknown parts of the state space and experimenting with different action sequences. The risk of exploration is that the agent might perform poorly in the short term by trying actions that turn out to be suboptimal. However, the benefit of exploration is that the agent might discover superior strategies that it would never find by sticking to known approaches.
Exploitation
Exploitation involves using the current best-known strategy based on what the agent has learned so far. This means maximizing immediate performance by choosing actions with the highest known Q-values. The risk of exploitation is that the agent might miss better strategies that exist but haven't been discovered yet, potentially getting stuck in a local optimum. However, the benefit of exploitation is consistent and reliable performance based on proven strategies.
Our agent uses the epsilon-greedy strategy to balance these competing objectives. This strategy works as follows: if a random number is less than epsilon, the agent chooses a random action to explore new possibilities. Otherwise, the agent chooses the best-known action to exploit current knowledge.
Epsilon Decay:
The epsilon parameter starts at one point zero, meaning one hundred percent exploration at the beginning of training. Over time, epsilon decays to zero point zero five, meaning five percent exploration after extensive training. The decay rate is zero point nine nine five per game, creating a gradual transition from exploration to exploitation.
This decay schedule creates a natural learning progression. During early games, the agent engages in random exploration, discovering what actions are possible and what consequences they produce. During middle games, the agent gradually begins trusting its learned knowledge while still maintaining significant exploration. During late games, the agent mostly exploits its learned policy with only occasional exploration to maintain adaptability.
๐ค Part 3: The AI Agent - Implementation Details
State Representation: Seeing the Game
One of the most critical design decisions in reinforcement learning is determining how the agent perceives the game. The state representation must capture essential information while remaining computationally manageable. If the state space is too large, learning becomes impossibly slow. If the state space is too small, the agent lacks the information needed to make good decisions.
The State Space:
The agent's state representation begins by discretizing the player's position into grid cells. The player's x-coordinate is divided by eighty pixels and rounded down, converting the continuous position into a discrete grid cell number. This discretization reduces the infinite continuous space of possible positions into a manageable set of discrete locations.
Next, the agent finds the closest enemy, which represents the most immediate threat. The agent calculates the distance to every enemy on screen and identifies which one is nearest to the player's ship. If no enemies are present, the state simply records the player position with a "none" indicator for the enemy information.
When an enemy is present, the agent discretizes the enemy's information using the same eighty-pixel grid system. The enemy's x-coordinate is divided by eighty and rounded down to get its horizontal grid position. The enemy's y-coordinate is divided by eighty and rounded down to get its vertical grid position. The relative x-position between the enemy and player is calculated by subtracting the player's x-coordinate from the enemy's x-coordinate and dividing by eighty.
Finally, these components are combined into a state string that uniquely identifies the current situation. For example, a state might be represented as "5_3_2_-2", meaning the player is at grid position five, the enemy is at grid position three horizontally and two vertically, and the enemy is two grid cells to the left of the player.
Why This Design?
The discretization using an eighty-pixel grid reduces the infinite continuous space to a manageable discrete state space. If the grid were too fine, there would be millions of possible states, making learning impossibly slow because each state would be visited very rarely. If the grid were too coarse, important information would be lost, preventing the agent from distinguishing between significantly different situations. The eighty-pixel grid represents a sweet spot for this particular game, providing enough granularity for effective decision-making without creating an unmanageably large state space.
Focusing on the closest enemy simplifies decision-making by reducing the amount of information the agent must process. In most situations, the closest enemy is the biggest immediate threat, so prioritizing this enemy makes strategic sense. This focus reduces state space complexity dramatically because the agent doesn't need to track every enemy simultaneously. By concentrating on the most relevant threat, the agent can learn faster and make more focused decisions.
Including relative position captures the spatial relationship between the player and the threat. This information helps the agent learn dodging behaviors because it can distinguish between enemies approaching from the left versus the right. The relative position also enables position-aware shooting strategies, allowing the agent to learn when it's well-positioned to hit enemies versus when it should focus on repositioning.
Example States:
A state string of "5_3_2_-2" indicates that the player is at grid position five, an enemy is at grid coordinates three comma two, and the enemy is two cells to the left of the player. A state string of "2_8_7_6" indicates that the player is at grid position two, an enemy is at grid coordinates eight comma seven, and the enemy is six cells to the right of the player. A state string of "5_none" indicates that the player is at grid position five and no enemies are currently present on the screen.
Action Space: What Can the AI Do?
The agent has four discrete actions available at each decision point, providing a simple but effective set of capabilities.
The first action is moving left, represented by action code zero. This action moves the player's ship leftward at twice the normal speed, allowing quick repositioning. This action is strategically used for dodging enemies approaching from the left and repositioning to better locations on the screen.
The second action is moving right, represented by action code one. This action moves the player's ship rightward at twice the normal speed, enabling rapid movement across the screen. This action is strategically used for dodging enemies approaching from the right and aligning the ship with enemies to improve shooting accuracy.
The third action is shooting, represented by action code two. This action fires a bullet upward toward enemies, respecting the cooldown timer to prevent excessive firing. This action is strategically used for destroying enemies to score points and clearing threats from the screen.
The fourth action is staying still, represented by action code three. This action involves no movement, keeping the player in the current position. This action is strategically used for conserving a good position when movement isn't necessary and waiting for the right moment to act.
Action Execution:
When the agent chooses action zero to move left, the game checks if the player's x-coordinate is greater than the left boundary. If the player has room to move left, the player's x-coordinate is decreased by the player's speed multiplied by two, creating rapid leftward movement.
When the agent chooses action one to move right, the game checks if the player's x-coordinate is less than the right boundary. If the player has room to move right, the player's x-coordinate is increased by the player's speed multiplied by two, creating rapid rightward movement.
When the agent chooses action two to shoot, the game calls the shoot function, which creates a new bullet and respects the cooldown timer to prevent firing too frequently.
When the agent chooses action three to stay, the game intentionally does nothing, maintaining the player's current position.
Why These Actions?
This action set is simple but effective, providing enough capabilities for complex strategies without overwhelming the learning process. The four actions are balanced in their capabilities, offering movement for positioning, offense for scoring, and patience for timing. The smaller action space means faster convergence because the agent has fewer options to evaluate in each state. These actions are also human-comparable, meaning they're similar to what human players can do, making the AI's performance directly comparable to human play.
Reward Function: Defining Success
The reward function is the heart of the learning process because it defines what "good" means to the AI. Our carefully crafted multi-component reward function encourages multiple desirable behaviors simultaneously.
Reward Components:
The reward calculation begins by initializing the reward to zero. Then, multiple components are added or subtracted based on different aspects of the agent's performance.
The first component is the survival reward, which adds one point to the reward for every frame the agent stays alive. This constant positive reward teaches the agent that staying alive is fundamentally good, even when nothing else is happening. This baseline reward ensures the agent learns that survival itself has value.
The second component is the score reward, which calculates the difference between the current score and the last recorded score. This score difference is multiplied by five and added to the reward. This strongly rewards destroying enemies because each enemy destruction increases the score, which in turn provides a large positive reward. The five-times multiplier makes scoring the primary objective, encouraging aggressive play.
The third component is the danger penalty, which penalizes the agent for being too close to enemies. For each enemy on the screen, the agent calculates the distance between the player and that enemy. If the distance is less than eighty pixels, a penalty is calculated as eighty minus the distance, divided by forty. This penalty increases as the agent gets closer to enemies, creating a "safety bubble" around the player. The accumulated danger penalty from all nearby enemies is subtracted from the reward, encouraging evasive maneuvers.
The fourth component is the aggression reward, which adds zero point two times the number of bullets currently on screen to the reward. This encourages active shooting rather than passive play, rewarding the agent for maintaining offensive pressure on enemies.
Finally, the total reward is returned, combining all these components into a single feedback signal.
Reward Design Philosophy:
The survival baseline of plus one per frame teaches the agent that staying alive is fundamentally good, providing a constant positive signal for continued existence. The primary objective is encoded through the five-times multiplier on score increases, making destroying enemies the main goal that dominates other considerations. The safety incentive creates a "safety bubble" around the player through the danger penalty, encouraging evasive maneuvers that keep the agent away from immediate threats. The aggressive play bonus provides a small reward for having bullets on screen, encouraging active engagement rather than purely defensive play.
Balancing Act:
The reward function creates interesting trade-offs that lead to emergent strategic behavior. The agent must decide whether to move closer to shoot an enemy, which involves risk, or stay safe in a defensive position, which involves caution. The agent must balance shooting constantly to maximize the aggression reward against conserving bullets for better opportunities. The agent must decide whether to chase power-ups, which represent opportunities for extra lives, or maintain a defensive position, which represents safety from immediate threats. These tensions create emergent strategic behavior as the agent learns optimal balance points through experience.
The Q-Table: The Agent's Memory
The Q-table is a data structure that stores all the agent's learned knowledge. Each entry in the Q-table maps a state-action pair to its estimated quality, which is the Q-value representing how good that action is in that state.
Structure:
The Q-table is implemented as a Map, which is a hash table data structure. Each key in the map is a string combining the state and action, such as "5_3_2_-2_0" for moving left from state "5_3_2_-2". The corresponding value is the Q-value, which might be negative for bad actions, near zero for neutral actions, or positive for good actions.
For example, the entry "5_3_2_-2_0" might map to negative one point two, indicating that moving left from this state tends to lead to poor outcomes. The entry "5_3_2_-2_1" might map to three point seven, indicating that moving right from this state tends to lead to moderately good outcomes. The entry "5_3_2_-2_2" might map to eight point four, indicating that shooting from this state tends to lead to very good outcomes, making it the best action. The entry "5_3_2_-2_3" might map to zero point five, indicating that staying still from this state tends to lead to slightly positive outcomes.
The Q-table contains thousands of such entries after extensive training, with each entry representing a piece of learned knowledge about the game.
How It Grows:
The Q-table starts completely empty, representing the agent's initial state of having no knowledge about the game. Each time the agent encounters a new state-action pair, that pair gets initialized to a Q-value of zero, representing neutral expectations. As the agent gains experience, these values are updated through the Q-learning formula, gradually becoming more accurate. After many games, the Q-table contains thousands of entries covering the most common and important game situations. This growing collection of entries represents the agent's accumulated wisdom about how to play Star Defender effectively.
Memory Efficiency:
Using a Map data structure provides several efficiency benefits. The map only stores state-action pairs that have actually been visited during training, avoiding wasted memory on impossible or extremely rare states. The lookup time for retrieving Q-values is constant on average, meaning the agent can quickly access its knowledge. The Q-table grows organically with experience, expanding naturally as the agent explores more of the game's state space.
Learning Parameters: Fine-Tuning the AI
The agent's learning behavior is controlled by several key hyperparameters that determine how quickly it learns, how much it values future rewards, and how it balances exploration with exploitation.
The learning rate, denoted as alpha, is set to zero point one. This parameter determines how fast the agent learns from new experiences. A moderate value like zero point one provides stable learning that responds to new information without being overly volatile.
The discount factor, denoted as gamma, is set to zero point ninety-five. This parameter determines how much the agent values future rewards compared to immediate rewards. A high value like zero point ninety-five encourages long-term planning by making future rewards almost as important as immediate rewards.
The initial epsilon is set to one point zero, meaning the agent starts with one hundred percent exploration. This ensures the agent thoroughly explores the state space at the beginning of training, discovering what actions are possible and what consequences they produce.
The minimum epsilon is set to zero point zero five, meaning the agent maintains five percent exploration even after extensive training. This ensures the agent always maintains some exploration to adapt to any changes or discover new strategies.
The epsilon decay rate is set to zero point nine nine five, meaning epsilon is multiplied by this factor after each game. This creates a gradual transition from exploration to exploitation over hundreds of games.
The decision frequency is set to every three frames, meaning the AI makes a new decision twenty times per second. This balances responsiveness with computational efficiency, giving actions time to execute while avoiding overwhelming the system with decisions.
Why These Values?
The learning rate of zero point one is not too aggressive, which would cause unstable learning with wild swings in Q-values, and not too conservative, which would cause extremely slow learning. This moderate value allows the agent to learn at a reasonable pace while maintaining stability.
The discount factor of zero point ninety-five values future rewards highly, encouraging strategic play that considers long-term consequences over short-term gains. This helps the agent learn that surviving longer leads to more opportunities to score points.
The epsilon decay to zero point zero five ensures the agent always maintains five percent exploration even after extensive training. This ongoing exploration allows the agent to adapt to any changes in the game and prevents it from becoming completely rigid in its strategy.
The decision frequency of every three frames gives actions time to execute and show their effects before making a new decision. This also reduces computational load by avoiding the need to make decisions every single frame, which would be sixty times per second.
⚙️ Part 4: Integration - How Game and AI Work Together
The Unified Game Loop
Both human and AI modes share the same game engine, with the control mechanism being the only difference between them. This unified architecture ensures that the AI learns to play the exact same game that humans play, making performance comparisons meaningful and fair.
Game Loop Structure:
The game loop runs at sixty frames per second, executing the following steps in sequence for each frame.
First, the loop handles input. If the mode is set to human, the game processes keyboard input from the arrow keys and spacebar. If the mode is set to AI and the current frame number is divisible by three, the game processes an AI decision. This conditional execution ensures AI decisions occur every three frames rather than every frame.
Second, the loop updates physics and game logic. This includes updating bullet positions as they travel upward, updating enemy positions as they descend and wobble, updating power-up positions as they fall, and updating particle positions as they disperse and fade.
Third, the loop performs collision detection. This includes checking for collisions between bullets and enemies, which result in enemy destruction and score increases. It also checks for collisions between the player and enemies, which result in life loss. Additionally, it checks for collisions between the player and power-ups, which result in gaining extra lives.
Fourth, the loop updates game state. This includes updating the score based on destroyed enemies, checking if the game should end due to the player running out of lives, and updating various counters and timers.
Fifth, the loop renders everything to the canvas. This includes drawing the background stars, drawing all game objects including the player, enemies, bullets, power-ups, and particles, and drawing the game over screen if applicable.
Sixth, the loop continues by calling requestAnimationFrame to schedule the next iteration. This creates a continuous loop that runs smoothly at sixty frames per second.
The AI Decision Cycle
Every three frames, which translates to twenty times per second, the AI makes a decision following a structured process.
Step-by-Step Process:
On frames zero, three, six, nine, and so on, the AI decision cycle begins. First, the agent perceives the environment by getting the current player position, finding the closest enemy, calculating relative positions between the player and threats, and creating a state string that uniquely identifies the current situation.
Second, the agent learns from its previous action. If a previous state and action exist, the agent calculates the reward received from the last action by considering survival time, score changes, danger levels, and aggression. The agent then updates the Q-value for the previous state-action pair using the Q-learning formula, incorporating the immediate reward and the estimated future value from the current state.
Third, the agent chooses a new action. The agent generates a random number between zero and one. If this number is less than epsilon, the agent chooses a random action to explore new possibilities. If the number is greater than or equal to epsilon, the agent chooses the action with the best Q-value to exploit current knowledge.
Fourth, the agent executes the chosen action. Depending on the action selected, the agent moves the player left or right, fires a bullet upward, or stays still in the current position.
Fifth, the agent stores the current state and action for use in the next learning cycle. This allows the agent to learn from the consequences of this action when the next decision cycle occurs.
Between Decision Frames:
On frames one, two, four, five, seven, eight, and other non-decision frames, the game continues running normally. The previous action continues executing, meaning if the agent chose to move left, the player continues moving left. All physics calculations continue, including bullet movement, enemy movement, and particle updates. Collision detection continues checking for impacts between game objects. Rendering continues drawing everything to the screen. However, the AI doesn't make new decisions during these frames, allowing the previous action to execute and show its effects.
This approach creates smooth gameplay while giving the AI time to see the consequences of its actions before making new decisions.
Continuous Training Loop
Unlike human players who typically play one game at a time with breaks between sessions, the AI can train indefinitely through an automatic restart mechanism.
Training Cycle:
When a game starts, the game state is initialized with a score of zero and lives set to three. The agent begins playing the game, making decisions every three frames and learning from the consequences of those decisions. During gameplay, the agent continuously updates its Q-table based on the rewards it receives, gradually refining its understanding of which actions work well in different situations.
When the game ends because the player has lost all lives, several important processes occur. First, the final score is recorded and stored in the agent's score history. Second, statistics are updated, including incrementing the games completed counter by one, adding the final score to the score history array, and updating the best score if the current game's score exceeded the previous record. Third, the epsilon value decays by being multiplied by the decay factor of zero point nine nine five, gradually reducing the exploration rate. Fourth, the user interface displays are updated to show the new statistics, including games completed, current epsilon value, best score, and average score over the last ten games.
After a brief pause of one hundred milliseconds to allow the UI to update, the game automatically restarts if the mode is still set to AI training. This automatic restart enables the AI to play hundreds or even thousands of games without any human intervention. The continuous training loop allows the agent to accumulate vast amounts of experience, far exceeding what would be practical for human players. Over time, this extensive training enables the agent to discover subtle patterns and strategies that might not be immediately obvious.
The automatic restart mechanism is crucial for effective reinforcement learning because the agent needs many examples to learn effectively. While a human might learn basic game mechanics in just a few games, the AI requires hundreds of games to build a comprehensive Q-table that covers most common situations. The continuous training loop makes this extensive training practical and efficient.
๐บ Part 5: What Viewers See On Screen
Main Game Canvas
The main game canvas measures eight hundred pixels wide by six hundred pixels tall and serves as the primary visual focus of the application. This canvas displays all game action and is where both human players and the AI agent interact with the game world.
Visual Layers:
The visual presentation consists of multiple layers rendered from bottom to top, creating a rich and engaging display.
The background layer forms the foundation of the visual presentation. It features a solid black space backdrop that provides high contrast for game elements. One hundred white stars scroll downward at varying speeds, creating a parallax effect that simulates traveling through space. These stars create depth and motion even when the player's ship is stationary, enhancing the sense of being in a dynamic space environment.
The trail effects layer shows the player's movement history. Fading cyan squares appear behind the player's ship, gradually becoming more transparent over time. These trails show the recent movement history of the player, creating a visual record of where the ship has been. The trails add visual polish to the game and help viewers track the player's movement patterns, which is particularly useful when observing the AI's behavior.
The player ship occupies the next layer. The cyan triangular spacecraft is positioned in the bottom-center region of the screen. The ship features a glowing neon effect created through shadow blur rendering, making it stand out prominently against the dark background. The triangular shape points upward, clearly indicating the direction the ship faces and where bullets will travel.
The enemies layer contains all hostile spacecraft. Red, orange, and magenta circles represent the three different enemy types, each with distinct sizes corresponding to their characteristics. Each enemy features glowing auras that make them highly visible and create the neon aesthetic. The enemies follow wobbling descent patterns, oscillating horizontally as they move downward, creating dynamic and unpredictable movement that challenges both human and AI players.
The bullets layer shows all projectiles currently in flight. Yellow rectangles represent individual bullets traveling upward toward enemies. Each bullet features bright glow trails that make them easy to track visually. The bullets move upward rapidly at ten units per frame, creating a clear offensive capability for the player.
The power-ups layer displays collectible items. Green rotating squares with a plus symbol in the center represent extra life power-ups. These power-ups fall slowly down the screen, giving players time to position themselves for collection. The rotation animation makes power-ups visually distinct and draws attention to these valuable items.
The particles layer contains all explosion and effect particles. These particles appear in various colors based on their source, including yellow for shooting effects, enemy colors for destruction effects, white for collision effects, and green for power-up collection effects. The particles fade out over time, creating temporary visual flourishes that provide feedback for player actions.
The game over screen appears as the topmost layer when applicable. A semi-transparent black overlay dims the entire game area, drawing focus to the game over message. The text "GAME OVER" appears in glowing red with a prominent shadow effect, making it impossible to miss. The final score is displayed below the game over message, showing the player's performance. For human players, restart instructions appear, prompting them to press Enter to play again.
Header Panel
The header panel spans the full width of the game container and displays critical game information in an organized layout.
Left Side - Score Information:
The left side of the header contains three score-related displays arranged horizontally with spacing between them. The first display shows the current score, which updates in real-time as enemies are destroyed. The label "SCORE" appears in small cyan text above the numerical value, which is displayed in large white text with a cyan glow effect. This score starts at zero at the beginning of each game and increases by ten, fifteen, or twenty points depending on which enemy type is destroyed.
The second display shows the high score, which represents the best performance across all game sessions. The label "HIGH SCORE" appears in small cyan text above the numerical value, which is displayed in large white text with a cyan glow effect. This high score persists across multiple games, providing a long-term goal for players to surpass. The high score updates immediately whenever the current score exceeds it, providing instant feedback when a new record is achieved.
The third display shows the remaining lives, indicating how many more hits the player can sustain before game over. The label "LIVES" appears in small cyan text above the numerical value, which is displayed in large white text with a cyan glow effect. This value starts at three at the beginning of each game and decreases by one each time the player collides with an enemy. When the lives counter reaches zero, the game ends.
All three displays feature cyan neon borders and glow effects that match the game's retro-futuristic aesthetic. The semi-transparent black background ensures the text remains readable against any background elements.
Right Side - Mode Indicator:
The right side of the header contains a mode indicator that shows the current play state. This indicator changes appearance and text based on what's currently happening in the application.
When the AI is actively playing, the indicator displays a robot emoji followed by the text "AI PLAYING" in capital letters. The background becomes a semi-transparent green, and the border changes to bright green. Most notably, the indicator features a pulsing animation that alternates the opacity between one hundred percent and sixty percent over a one point five second cycle. This pulsing effect immediately draws attention and clearly communicates that the AI is actively training.
When a human is playing, the indicator displays a person emoji followed by the text "HUMAN PLAYING" in capital letters. The background becomes a semi-transparent magenta, and the border changes to bright magenta. Unlike the AI mode, there is no pulsing animation, creating a static display that indicates human control.
When no game is active, the indicator simply displays the text "IDLE" in capital letters. The background and border use gray tones, and there is no animation. This neutral appearance clearly communicates that the application is waiting for user input to begin a game.
Control Buttons
Below the game canvas, four buttons provide control over the application's behavior. These buttons are arranged horizontally with even spacing between them.
The first button is labeled "Play (Human)" and initiates a human-controlled game session. When clicked, this button starts a new game where the player controls the ship using arrow keys for movement and the spacebar for shooting. After clicking this button, it becomes disabled along with the "Start AI Training" button, preventing multiple game modes from running simultaneously. The "Stop AI" button remains disabled because there's no AI training to stop during human play.
The second button is labeled "Start AI Training" and initiates an AI-controlled training session. When clicked, this button starts the continuous training loop where the AI plays repeatedly, learning from each game. After clicking this button, it becomes disabled along with the "Play (Human)" button, preventing mode conflicts. The "Stop AI" button becomes enabled, allowing the user to halt training at any time.
The third button is labeled "Stop AI" and halts any ongoing AI training. This button is disabled by default and only becomes enabled when AI training is active. When clicked, it stops the automatic restart loop, sets the game mode to idle, and re-enables the "Play (Human)" and "Start AI Training" buttons. This allows users to pause training to examine statistics or switch to human play mode.
The fourth button is labeled "Reset High Score" and clears all accumulated data. When clicked, this button first displays a confirmation dialog asking "Reset high score and AI learning data?" to prevent accidental resets. If the user confirms, the high score is reset to zero, the entire Q-table is cleared, epsilon is reset to one point zero, games completed is reset to zero, the score history is cleared, and the best AI score is reset to zero. All UI displays update immediately to reflect these changes. This reset functionality is useful for starting fresh training runs or clearing data from previous experiments.
All four buttons feature gradient purple backgrounds that transition from lighter to darker shades, creating visual depth. The buttons have cyan neon borders that match the game's color scheme. Glow effects surround each button, making them prominent and easy to locate. When the user hovers their mouse over an enabled button, the button lifts up slightly through a transform animation, and the glow effect intensifies. When a button is clicked, it returns to its original position, providing tactile feedback. Disabled buttons appear with reduced opacity at fifty percent and display a "not allowed" cursor, clearly indicating they cannot be clicked in the current state.
RL Agent Statistics Panel
Below the control buttons, a comprehensive dashboard displays detailed information about the AI agent's learning progress. This panel is always visible, allowing viewers to monitor the AI's development even when it's not actively playing.
The panel header displays a robot emoji followed by the text "REINFORCEMENT LEARNING AGENT" in large magenta text. This title is centered and uses uppercase letters to create a prominent section header. The entire panel features a semi-transparent black background with a magenta neon border and glow effect, maintaining visual consistency with the rest of the interface.
The statistics are arranged in a responsive grid layout that adapts to screen size. On wider screens, the statistics appear in multiple columns, while on narrower screens they stack vertically. Each statistic occupies its own cell with a semi-transparent magenta background and a thin magenta border.
Games Completed:
The first statistic shows the total number of games the AI has played since the application loaded or since the last reset. The label "Games Completed" appears in small magenta text above the numerical value. The value is displayed in large white bold text, starting at zero and incrementing by one after each game the AI completes. This metric provides a clear indication of how much training the AI has undergone. In the early stages, this number increases rapidly as the AI plays short games. As the AI improves and survives longer, games take more time, so the counter increases more slowly.
Epsilon (Exploration):
The second statistic shows the current exploration rate, which determines how often the AI tries random actions versus exploiting its learned knowledge. The label "Epsilon (Exploration)" appears in small magenta text above the numerical value. The value is displayed in large white bold text showing two decimal places, starting at one point zero zero and gradually decreasing toward zero point zero five.
Below the numerical value, a progress bar provides a visual representation of the epsilon value. The progress bar consists of a container with a semi-transparent white background and a fill element that uses a gradient from magenta to cyan. The width of the fill element corresponds to the epsilon value, starting at one hundred percent width and gradually shrinking as epsilon decays. This visual representation makes it easy to see at a glance how far the AI has progressed from pure exploration toward primarily exploitation.
AI Best Score:
The third statistic shows the highest score the AI has achieved across all games. The label "AI Best Score" appears in small magenta text above the numerical value. The value is displayed in large white bold text, starting at zero and updating whenever the AI achieves a new personal record. This metric provides a clear measure of the AI's peak performance capability. Viewers can compare this value to their own high scores to see how the AI's best performance compares to human play. As training progresses, this value typically increases, demonstrating the AI's improvement over time.
Average Score (Last 10):
The fourth statistic shows the average score over the AI's last ten games. The label "Avg Score (Last 10)" appears in small magenta text above the numerical value. The value is displayed in large white bold text, calculated by summing the scores from the last ten games and dividing by ten, rounding to the nearest integer. This rolling average provides insight into the AI's current consistent performance level, as opposed to its peak performance. A rising average indicates the AI is improving its typical performance, while a stable average suggests the AI has reached a plateau in its learning. This metric is particularly useful for identifying when the AI has achieved consistent mastery versus occasional lucky high scores.
Learning Rate:
The fifth statistic shows the learning rate parameter, which controls how quickly the AI updates its Q-values based on new experiences. The label "Learning Rate" appears in small magenta text above the numerical value. The value is displayed in large white bold text showing two decimal places, consistently showing zero point one zero throughout training. This parameter remains constant in our implementation, providing stable learning dynamics. Displaying this value helps viewers understand that the AI's learning speed is controlled and consistent, rather than varying over time.
Q-Table Size:
The sixth statistic shows the number of state-action pairs the AI has learned about. The label "Q-Table Size" appears in small magenta text above the numerical value. The value is displayed in large white bold text, starting at zero and growing as the AI encounters new situations. Each unique combination of state and action that the AI experiences adds one entry to the Q-table. This metric provides insight into how much knowledge the AI has accumulated. In early training, this number grows rapidly as the AI discovers many new situations. As training progresses, growth slows because the AI encounters fewer truly novel situations. A large Q-table size indicates comprehensive coverage of the game's state space.
Visual Feedback During AI Play
As viewers watch the AI train, they can observe distinct behavioral patterns that change as learning progresses. These observable behaviors provide insight into the AI's internal learning process.
Early Games (High Epsilon):
During the first fifty to one hundred games, when epsilon is high, the AI's behavior appears erratic and seemingly random. The ship moves left and right without apparent purpose, frequently changing direction mid-movement. Collisions with enemies occur frequently because the AI hasn't learned to avoid them. Shooting happens at random times, often when no enemies are nearby or when enemies are too far away to hit. Scores during this phase are typically very low, often ranging from zero to fifty points, with many games ending almost immediately. To human observers, the AI looks "confused" or "drunk," moving without any coherent strategy. However, this chaotic behavior is actually productive exploration, allowing the AI to discover the consequences of different actions in different situations.
Middle Games (Moderate Epsilon):
During games fifty to two hundred, when epsilon has decayed to moderate levels, the AI's behavior begins to show signs of intentionality. Movement becomes more purposeful, with the ship sometimes moving away from approaching enemies rather than randomly. Some dodging behavior emerges, particularly when enemies are very close, showing the AI has learned that proximity to enemies is dangerous. Shot timing improves, with more bullets fired when enemies are actually present and within range. Scores during this phase are moderate, typically ranging from fifty to one hundred fifty points. The AI's play shows occasional good decisions mixed with mistakes, creating an inconsistent but improving performance. Viewers can see the AI sometimes making smart moves followed by seemingly random actions, reflecting the balance between exploitation and exploration.
Late Games (Low Epsilon):
After three hundred or more games, when epsilon has decayed to low levels, the AI's behavior appears smooth and strategic. Movement becomes fluid and deliberate, with the ship positioning itself carefully relative to enemies. Consistent enemy dodging is evident, with the AI anticipating enemy positions and moving early to avoid collisions. Shot timing becomes well-calibrated, with bullets fired when enemies are in favorable positions for hits. Scores during this phase are high, typically ranging from one hundred fifty to three hundred or more points. The AI's play looks "skilled" and intentional, with movements that appear purposeful and effective. Viewers familiar with the game can recognize strategies similar to what experienced human players use, such as maintaining central positions, moving early to avoid threats, and timing shots for maximum effectiveness.
๐ฌ Part 6: The Learning Process in Detail
Phase 1: Random Exploration (Games 1-50)
During the first phase of training, the AI is essentially learning the basics of the game through random experimentation.
What's Happening:
The epsilon value during this phase ranges from approximately one point zero down to zero point six, meaning the AI chooses random actions between sixty and one hundred percent of the time. Almost all actions are random, with very little exploitation of learned knowledge. The Q-table is rapidly growing as the AI encounters new state-action pairs for the first time. The AI is discovering basic cause-and-effect relationships, such as what happens when it moves left, what happens when it shoots, and why it loses lives.
Observable Behavior:
The ship moves randomly across the screen without apparent strategy or purpose. Shooting occurs at random times, often when no enemies are present or when enemies are too far away to be hit. The AI frequently crashes into enemies because it hasn't learned to avoid them. Scores are typically very low during this phase, usually ranging from zero to thirty points, with many games ending within seconds.
Learning Focus:
The AI is asking and answering fundamental questions about the game mechanics. It's learning what happens when it moves left by observing that the ship's position changes and sometimes collisions occur. It's learning what happens when it shoots by observing that bullets appear and sometimes enemies disappear. It's learning why it loses lives by experiencing collisions with enemies and observing the life counter decrease. Through these experiences, the AI is building initial Q-value estimates that will serve as the foundation for more sophisticated learning.
Q-Table Growth:
The Q-table grows from zero entries to approximately five hundred entries during this phase. These entries cover common game situations that the AI encounters frequently, such as enemies at various positions and distances. Many Q-values remain near zero during this phase because the AI hasn't experienced enough examples to develop strong preferences. The Q-table is establishing a basic structure that will be refined in later phases.
Phase 2: Pattern Recognition (Games 51-150)
During the second phase of training, the AI begins to recognize patterns and develop basic strategies.
What's Happening:
The epsilon value during this phase ranges from approximately zero point six down to zero point three, meaning the AI is now exploiting learned knowledge thirty to forty percent of the time. There's a productive mix of exploration and exploitation, with the AI trying new things while also using strategies it has found effective. Q-values are becoming more differentiated, with clear differences emerging between good and bad actions in specific situations. The AI is recognizing dangerous versus safe situations, learning to distinguish between states where enemies are close versus far away.
Observable Behavior:
Occasional intentional dodges become visible, where the ship moves away from approaching enemies rather than moving randomly. Some enemies are successfully destroyed as the AI learns to time its shots better. However, many mistakes still occur, with the AI sometimes moving into danger or failing to shoot when opportunities arise. Scores during this phase are typically moderate, ranging from thirty to one hundred points, showing clear improvement over the random exploration phase.
Learning Focus:
The AI is learning important strategic principles. It's discovering that moving away from enemies is good because it reduces the danger penalty and helps maintain lives. It's learning that shooting when enemies are ahead works well because it leads to score increases and positive rewards. It's recognizing that staying in the middle of the screen is safer than the edges because it provides more options for dodging in either direction. Through these insights, the AI is refining its reward associations and building a more sophisticated understanding of effective play.
Q-Table Growth:
The Q-table grows from approximately five hundred to two thousand entries during this phase. This expansion represents wider coverage of the game's state space as the AI explores more situations. Clear positive and negative Q-values are emerging, with some actions showing consistently high values and others showing consistently low values. The differentiation in Q-values enables the AI to make increasingly informed decisions.
Phase 3: Strategy Development (Games 151-300)
During the third phase of training, the AI develops sophisticated strategies and achieves consistent performance.
What's Happening:
The epsilon value during this phase ranges from approximately zero point three down to zero point fifteen, meaning the AI is now exploiting learned knowledge eighty-five to seventy percent of the time. The AI is mostly using exploitation with some exploration to refine its strategies. Sophisticated strategies are emerging, such as predictive positioning and aggressive shooting when safe. Performance becomes more consistent, with fewer extremely low scores and more reliably good scores.
Observable Behavior:
Deliberate positioning becomes evident, with the ship moving to specific locations that provide strategic advantages. Predictive dodging appears, where the AI moves early to avoid enemies before they become immediate threats. Aggressive shooting when safe shows the AI has learned to balance offense and defense effectively. Scores during this phase typically range from one hundred to two hundred points, demonstrating solid competence at the game.
Learning Focus:
The AI is optimizing action sequences and developing multi-step strategies. It's learning that it should position itself under enemies to shoot them effectively, maximizing hit probability. It's discovering that it should move early to avoid collisions rather than waiting until enemies are dangerously close. It's recognizing that shooting frequently is better than waiting because it creates more opportunities to destroy enemies. These insights represent sophisticated strategic thinking that goes beyond simple stimulus-response associations.
Q-Table Growth:
The Q-table grows from approximately two thousand to four thousand entries during this phase. This expansion covers edge cases and unusual situations that occur less frequently. The Q-values become fine-tuned through repeated experiences, with values converging toward their optimal levels. The Q-table now represents a comprehensive strategy for playing the game effectively.
Phase 4: Mastery (Games 300+)
During the fourth phase of training, the AI achieves mastery and maintains near-optimal performance.
What's Happening:
The epsilon value during this phase ranges from approximately zero point fifteen down to zero point zero five, meaning the AI is exploiting learned knowledge ninety-five to eighty-five percent of the time. The AI is primarily using exploitation, relying on its well-developed strategy. The policy is near-optimal, meaning the AI is making decisions that are close to the best possible choices in most situations. Performance is stable and high, with consistently good scores and rare catastrophic failures.
Observable Behavior:
Smooth and confident movement characterizes the AI's play, with deliberate actions that flow naturally from one to the next. The AI rarely gets hit by enemies, demonstrating effective dodging and positioning skills. The enemy destruction rate is high, with the AI successfully hitting a large percentage of enemies. Scores during this phase typically range from one hundred fifty to three hundred or more points, demonstrating mastery of the game.
Learning Focus:
The AI is fine-tuning existing knowledge rather than discovering fundamentally new strategies. It's adapting to rare situations that occur infrequently, filling in gaps in its knowledge. The five percent exploration rate maintains adaptability, ensuring the AI doesn't become completely rigid in its strategy. The AI is polishing optimal strategies, making small adjustments that incrementally improve performance.
Q-Table Growth:
The Q-table grows from approximately four thousand to six thousand or more entries during this phase. This growth represents comprehensive coverage of the game's state space, including rare and unusual situations. The Q-values are mature and stable, changing only slightly with new experiences. The Q-table now represents a complete and sophisticated understanding of how to play Star Defender effectively.
๐ Part 7: Comparing Human vs. AI Performance
Human Players
Human players bring unique strengths and weaknesses to the game that differ fundamentally from the AI's capabilities.
Humans possess instant pattern recognition abilities that allow them to recognize enemy formations and dangerous situations immediately without needing hundreds of examples. Humans have intuitive spatial reasoning that enables them to judge distances, predict trajectories, and plan movements naturally. Humans can learn from single examples, understanding that crashing into enemies is bad after experiencing it just once or twice. Humans are adaptable to new situations immediately, able to handle unexpected scenarios without prior specific training. Humans understand game goals without explicit rewards, intuitively knowing that surviving longer and scoring more points is desirable.
Disadvantages:
Humans have reaction time limits of approximately two hundred milliseconds between perceiving a threat and responding to it. Humans experience fatigue and attention lapses, with performance degrading during long play sessions. Human performance is inconsistent, varying based on mood, focus level, and other psychological factors. Humans have limited practice time, typically playing for minutes or hours rather than the continuous training the AI can achieve. Humans have emotional responses such as frustration and overconfidence that can negatively impact decision-making.
Typical Learning Curve:
During their first game, human players typically score twenty to fifty points while learning the controls and basic mechanics. By their tenth game, human players usually achieve one hundred to one hundred fifty points, demonstrating basic competence with movement and shooting. By their fiftieth game, skilled human players can achieve two hundred to three hundred points through practiced play. Human performance typically plateaus around three hundred to four hundred points, representing the practical limit for most players given reaction time constraints and the game's difficulty.
AI Agent
The AI agent brings different strengths and weaknesses that complement and contrast with human capabilities.
Advantages:
The AI has perfect consistency with no fatigue, maintaining the same level of performance indefinitely without degradation. The AI can train twenty-four hours a day, seven days a week, accumulating far more experience than any human player could achieve. The AI has exact timing with no reaction delay, able to execute actions in the same frame it makes decisions. The AI is emotionless, never experiencing tilt, frustration, or overconfidence that could impair decision-making. The AI engages in systematic exploration of all strategies, thoroughly testing different approaches rather than relying on intuition.
Disadvantages:
The AI has slow initial learning, needing many examples to learn concepts that humans grasp immediately. The AI is limited by state representation, only able to consider information explicitly included in its state encoding. The AI cannot generalize beyond training scenarios, struggling with situations that differ significantly from its training experience. The AI requires careful reward engineering, needing humans to define what "good" means through the reward function. The AI has no intuitive understanding, only knowing statistical correlations between states, actions, and rewards without understanding causation.
Typical Learning Curve:
During games one through fifty, the AI typically scores zero to thirty points while engaging in random exploration. During games fifty through one hundred fifty, the AI scores thirty to one hundred points as it recognizes patterns and develops basic strategies. During games one hundred fifty through three hundred, the AI scores one hundred to two hundred points as it develops sophisticated strategies. After game three hundred, the AI scores one hundred fifty to three hundred or more points, demonstrating mastery of the game.
Interesting Observations
AI Discovers Non-Intuitive Strategies:
Sometimes the AI finds strategies that humans wouldn't naturally try. For example, the AI sometimes prefers staying at screen edges rather than the center, which humans typically avoid. The AI engages in constant shooting even without clear targets, implementing a suppressive fire strategy. The AI makes small and frequent adjustments rather than large movements, which differs from typical human play patterns that involve more dramatic repositioning.
Humans Adapt Faster:
A human player can learn that crashing into enemies is bad after just one or two games. The AI needs fifty to one hundred games to fully internalize this concept through statistical learning. This difference highlights the fundamental distinction between human causal reasoning and AI statistical learning.
AI Achieves Higher Consistency:
Once trained, the AI maintains consistent performance across all games, with scores clustering around its average performance level. Humans have good games and bad games depending on focus, mood, and fatigue, leading to higher performance variability.
Different "Understanding":
Humans understand why strategies work through causal reasoning, able to explain that dodging enemies prevents collisions which preserves lives. The AI only knows that strategies work through statistical correlation, recognizing that certain actions in certain states lead to higher rewards without understanding the underlying causation. Humans can explain their decisions using concepts like safety, opportunity, and risk. The AI just has Q-values that represent learned associations without conceptual understanding.
๐ Part 8: Educational Value & Key Takeaways
What This Demonstration Teaches
Reinforcement Learning is Trial-and-Error Learning:
Unlike supervised learning where the algorithm learns from labeled examples with correct answers provided, RL agents learn by doing and experiencing consequences. They make mistakes, receive feedback through rewards and penalties, and gradually improve through accumulated experience. This learning process is much more similar to how humans learn new skills, making RL particularly intuitive to understand through interactive demonstrations like Star Defender.
Reward Design is Critical:
The reward function shapes all behavior by defining what "good" and "bad" mean to the agent. Small changes in rewards can dramatically alter what the agent learns, potentially leading to unexpected or undesired behaviors. This highlights the importance of alignment between reward signals and desired outcomes, a critical consideration in real-world AI applications. The multi-component reward function in Star Defender demonstrates how complex objectives can be encoded through carefully balanced reward signals.
Exploration vs. Exploitation is Fundamental:
The epsilon-greedy strategy demonstrates a core RL dilemma: should you try something new that might be better, or stick with what you know works? This trade-off appears in many real-world scenarios beyond gaming, including business strategy, scientific research, and personal decision-making. The gradual decay from exploration to exploitation mirrors how humans often approach new challenges, starting with broad experimentation and gradually converging on proven strategies.
Learning Takes Time:
The AI needs hundreds of games to master what humans learn in dozens, highlighting the sample efficiency gap between human and machine learning. However, the AI's ability to train continuously without fatigue means it can eventually accumulate far more experience than any human. This demonstrates both the limitations and advantages of machine learning compared to human learning.
State Representation Matters:
The way the agent perceives the game fundamentally limits what it can learn. The discretization into eighty-pixel grid cells and focus on the closest enemy represent design choices that balance computational efficiency with information richness. Different state representations would lead to different learning outcomes, demonstrating how the interface between agent and environment shapes the learning process.
Emergent Behavior from Simple Rules:
The sophisticated strategies the AI develops emerge from the simple Q-learning algorithm combined with the reward function. No one programmed the AI to dodge enemies in specific ways or to position itself strategically. These behaviors emerged naturally through the learning process, demonstrating how complex intelligent behavior can arise from relatively simple learning rules.
Transparency Enables Understanding:
The visible statistics panel showing epsilon, Q-table size, and performance metrics provides transparency into the AI's learning process. This transparency helps viewers understand what's happening inside the "black box" of machine learning, demystifying AI and making it more accessible. The ability to watch the AI's behavior change over time provides intuitive insight into how learning progresses.
AI and Humans Excel Differently:
The comparison between human and AI performance reveals that intelligence comes in different forms. Humans excel at rapid learning, intuitive understanding, and flexible adaptation. AI excels at consistency, tireless practice, and systematic exploration. Neither approach is universally superior; each has strengths and weaknesses suited to different contexts.
Broader Implications
The principles demonstrated in Star Defender extend far beyond gaming. Reinforcement learning is used in robotics to teach robots to walk, grasp objects, and navigate environments. It's applied in autonomous vehicles to learn driving policies that balance safety and efficiency. It's used in resource management to optimize energy consumption, traffic flow, and supply chains. It's applied in personalized recommendations to learn user preferences and suggest relevant content. It's used in financial trading to develop strategies that adapt to changing market conditions.
The Star Defender demonstration provides an accessible entry point to understanding these broader applications. By watching an AI learn to play a simple game, viewers gain intuition about how AI systems learn in more complex domains. The transparent learning process, visible statistics, and comparable human performance make reinforcement learning concrete and understandable rather than abstract and mysterious.
Conclusion
Star Defender represents more than just a game or a technical demonstration. It serves as an educational tool that makes reinforcement learning tangible and accessible. By combining engaging gameplay with transparent AI learning, the application provides unique insights into how machines develop intelligent behavior through experience.
Viewers can observe the complete learning journey from random exploration to strategic mastery, gaining intuitive understanding of concepts like exploration-exploitation tradeoffs, reward shaping, and policy development. The ability to play the game themselves and compare their performance to the AI's creates a personal connection that enhances learning and engagement.
Whether you're a student learning about AI, a developer exploring reinforcement learning, or simply someone curious about how machines learn, Star Defender offers a window into the fascinating world of artificial intelligence. The application demonstrates that AI learning, while different from human learning, follows understandable principles that can be observed, analyzed, and appreciated through interactive demonstration.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Star Defender - RL Demo</title>
<style>
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Courier New', monospace;
background: linear-gradient(to bottom, #0a0a1a, #1a0a2e);
display: flex;
justify-content: center;
align-items: center;
min-height: 100vh;
color: #fff;
overflow: hidden;
}
.game-container {
position: relative;
max-width: 1200px;
width: 100%;
padding: 20px;
}
.header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 20px;
background: rgba(0, 0, 0, 0.5);
padding: 15px;
border-radius: 10px;
border: 2px solid #00ffff;
box-shadow: 0 0 20px rgba(0, 255, 255, 0.5);
}
.score-panel {
display: flex;
gap: 30px;
}
.score-item {
text-align: center;
}
.score-label {
font-size: 14px;
color: #00ffff;
text-transform: uppercase;
}
.score-value {
font-size: 28px;
font-weight: bold;
color: #fff;
text-shadow: 0 0 10px #00ffff;
}
.mode-indicator {
padding: 10px 20px;
background: rgba(255, 0, 255, 0.3);
border: 2px solid #ff00ff;
border-radius: 5px;
font-weight: bold;
text-transform: uppercase;
}
.mode-indicator.ai-active {
background: rgba(0, 255, 0, 0.3);
border-color: #00ff00;
animation: pulse 1.5s infinite;
}
canvas {
display: block;
background: #000;
border: 3px solid #00ffff;
border-radius: 10px;
box-shadow: 0 0 30px rgba(0, 255, 255, 0.7);
margin: 0 auto;
}
.controls {
display: flex;
justify-content: center;
gap: 20px;
margin-top: 20px;
}
button {
padding: 12px 30px;
font-size: 16px;
font-weight: bold;
color: #fff;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
border: 2px solid #00ffff;
border-radius: 8px;
cursor: pointer;
text-transform: uppercase;
transition: all 0.3s;
box-shadow: 0 0 15px rgba(0, 255, 255, 0.5);
}
button:hover {
transform: translateY(-2px);
box-shadow: 0 0 25px rgba(0, 255, 255, 0.8);
}
button:active {
transform: translateY(0);
}
button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.rl-panel {
background: rgba(0, 0, 0, 0.7);
padding: 20px;
border-radius: 10px;
border: 2px solid #ff00ff;
margin-top: 20px;
box-shadow: 0 0 20px rgba(255, 0, 255, 0.5);
}
.rl-title {
font-size: 20px;
color: #ff00ff;
margin-bottom: 15px;
text-align: center;
text-transform: uppercase;
}
.rl-stats {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
gap: 15px;
}
.rl-stat {
background: rgba(255, 0, 255, 0.1);
padding: 10px;
border-radius: 5px;
border: 1px solid #ff00ff;
}
.rl-stat-label {
font-size: 12px;
color: #ff00ff;
margin-bottom: 5px;
}
.rl-stat-value {
font-size: 20px;
font-weight: bold;
color: #fff;
}
.progress-bar {
width: 100%;
height: 10px;
background: rgba(255, 255, 255, 0.1);
border-radius: 5px;
overflow: hidden;
margin-top: 5px;
}
.progress-fill {
height: 100%;
background: linear-gradient(90deg, #ff00ff, #00ffff);
transition: width 0.3s;
}
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.6; }
}
.learning {
animation: pulse 2s infinite;
}
.instructions {
text-align: center;
margin-top: 10px;
color: #00ffff;
font-size: 14px;
}
</style>
</head>
<body>
<div class="game-container">
<div class="header">
<div class="score-panel">
<div class="score-item">
<div class="score-label">Score</div>
<div class="score-value" id="score">0</div>
</div>
<div class="score-item">
<div class="score-label">High Score</div>
<div class="score-value" id="highScore">0</div>
</div>
<div class="score-item">
<div class="score-label">Lives</div>
<div class="score-value" id="lives">3</div>
</div>
</div>
<div class="mode-indicator" id="modeIndicator">IDLE</div>
</div>
<canvas id="gameCanvas" width="800" height="600"></canvas>
<div class="instructions">
Human Mode: Arrow Keys = Move | Spacebar = Shoot
</div>
<div class="controls">
<button id="playBtn">Play (Human)</button>
<button id="aiBtn">Start AI Training</button>
<button id="stopBtn" disabled>Stop AI</button>
<button id="resetBtn">Reset High Score</button>
</div>
<div class="rl-panel">
<div class="rl-title">๐ค Reinforcement Learning Agent</div>
<div class="rl-stats">
<div class="rl-stat">
<div class="rl-stat-label">Games Completed</div>
<div class="rl-stat-value" id="gamesCompleted">0</div>
</div>
<div class="rl-stat">
<div class="rl-stat-label">Epsilon (Exploration)</div>
<div class="rl-stat-value" id="epsilon">1.00</div>
<div class="progress-bar">
<div class="progress-fill" id="epsilonBar" style="width: 100%"></div>
</div>
</div>
<div class="rl-stat">
<div class="rl-stat-label">AI Best Score</div>
<div class="rl-stat-value" id="aiBestScore">0</div>
</div>
<div class="rl-stat">
<div class="rl-stat-label">Avg Score (Last 10)</div>
<div class="rl-stat-value" id="avgScore">0</div>
</div>
<div class="rl-stat">
<div class="rl-stat-label">Learning Rate</div>
<div class="rl-stat-value" id="learningRate">0.10</div>
</div>
<div class="rl-stat">
<div class="rl-stat-label">Q-Table Size</div>
<div class="rl-stat-value" id="qTableSize">0</div>
</div>
</div>
</div>
</div>
<script>
const canvas = document.getElementById('gameCanvas');
const ctx = canvas.getContext('2d');
// Game State
let gameState = {
score: 0,
highScore: 0,
lives: 3,
gameOver: true, // Start as true, will be set to false when game starts
paused: false,
mode: 'idle', // 'idle', 'human', 'ai'
frame: 0,
lastScore: 0
};
// Player
const player = {
x: canvas.width / 2,
y: canvas.height - 80,
width: 40,
height: 40,
speed: 7,
color: '#00ffff',
trail: [],
lastShot: 0
};
// Game Objects
let bullets = [];
let enemies = [];
let particles = [];
let stars = [];
let powerUps = [];
// Input
const keys = {};
// Reinforcement Learning Agent
class RLAgent {
constructor() {
this.qTable = new Map();
this.epsilon = 1.0; // Exploration rate
this.epsilonMin = 0.05;
this.epsilonDecay = 0.995;
this.learningRate = 0.1;
this.discountFactor = 0.95;
this.gamesCompleted = 0;
this.scoreHistory = [];
this.bestScore = 0;
this.lastState = null;
this.lastAction = null;
this.lastScore = 0;
}
getState() {
// Discretize the game state
const playerPos = Math.floor(player.x / 80);
// Find closest enemy
let closestEnemy = null;
let minDist = Infinity;
enemies.forEach(enemy => {
const dist = Math.sqrt(
Math.pow(enemy.x - player.x, 2) +
Math.pow(enemy.y - player.y, 2)
);
if (dist < minDist) {
minDist = dist;
closestEnemy = enemy;
}
});
if (!closestEnemy) {
return `${playerPos}_none`;
}
const enemyX = Math.floor(closestEnemy.x / 80);
const enemyY = Math.floor(closestEnemy.y / 80);
const relativeX = Math.floor((closestEnemy.x - player.x) / 80);
return `${playerPos}_${enemyX}_${enemyY}_${relativeX}`;
}
getQValue(state, action) {
const key = `${state}_${action}`;
return this.qTable.get(key) || 0;
}
setQValue(state, action, value) {
const key = `${state}_${action}`;
this.qTable.set(key, value);
}
chooseAction(state) {
// Epsilon-greedy strategy
if (Math.random() < this.epsilon) {
// Exploration: random action
return Math.floor(Math.random() * 4); // 0: left, 1: right, 2: shoot, 3: stay
} else {
// Exploitation: best known action
let bestAction = 0;
let bestValue = this.getQValue(state, 0);
for (let action = 1; action < 4; action++) {
const value = this.getQValue(state, action);
if (value > bestValue) {
bestValue = value;
bestAction = action;
}
}
return bestAction;
}
}
learn(state, action, reward, nextState) {
const currentQ = this.getQValue(state, action);
// Find max Q value for next state
let maxNextQ = this.getQValue(nextState, 0);
for (let a = 1; a < 4; a++) {
maxNextQ = Math.max(maxNextQ, this.getQValue(nextState, a));
}
// Q-learning update
const newQ = currentQ + this.learningRate *
(reward + this.discountFactor * maxNextQ - currentQ);
this.setQValue(state, action, newQ);
}
executeAction(action) {
// 0: left, 1: right, 2: shoot, 3: stay
switch(action) {
case 0: // Move left
if (player.x > player.width / 2 + 20) {
player.x -= player.speed * 2;
}
break;
case 1: // Move right
if (player.x < canvas.width - player.width / 2 - 20) {
player.x += player.speed * 2;
}
break;
case 2: // Shoot
shoot();
break;
case 3: // Stay
// Do nothing
break;
}
}
calculateReward() {
let reward = 0;
// Reward for staying alive
reward += 1;
// Reward for score increase
const scoreDiff = gameState.score - this.lastScore;
reward += scoreDiff * 5;
this.lastScore = gameState.score;
// Penalty for being too close to enemies
let dangerPenalty = 0;
enemies.forEach(enemy => {
const dist = Math.sqrt(
Math.pow(enemy.x - player.x, 2) +
Math.pow(enemy.y - player.y, 2)
);
if (dist < 80) {
dangerPenalty += (80 - dist) / 40;
}
});
reward -= dangerPenalty;
// Reward for having bullets on screen (being aggressive)
reward += bullets.length * 0.2;
return reward;
}
gameEnded(finalScore) {
this.gamesCompleted++;
this.scoreHistory.push(finalScore);
if (this.scoreHistory.length > 10) {
this.scoreHistory.shift();
}
if (finalScore > this.bestScore) {
this.bestScore = finalScore;
}
// Decay epsilon
this.epsilon = Math.max(this.epsilonMin, this.epsilon * this.epsilonDecay);
// Update UI
this.updateUI();
// Auto-restart if in AI mode
if (gameState.mode === 'ai') {
setTimeout(() => {
startGame('ai');
}, 100);
}
}
updateUI() {
document.getElementById('gamesCompleted').textContent = this.gamesCompleted;
document.getElementById('epsilon').textContent = this.epsilon.toFixed(2);
document.getElementById('epsilonBar').style.width = (this.epsilon * 100) + '%';
document.getElementById('aiBestScore').textContent = this.bestScore;
const avgScore = this.scoreHistory.length > 0
? Math.round(this.scoreHistory.reduce((a, b) => a + b, 0) / this.scoreHistory.length)
: 0;
document.getElementById('avgScore').textContent = avgScore;
document.getElementById('learningRate').textContent = this.learningRate.toFixed(2);
document.getElementById('qTableSize').textContent = this.qTable.size;
}
}
const agent = new RLAgent();
// Initialize stars
for (let i = 0; i < 100; i++) {
stars.push({
x: Math.random() * canvas.width,
y: Math.random() * canvas.height,
size: Math.random() * 2,
speed: Math.random() * 2 + 0.5,
opacity: Math.random()
});
}
// Particle system
function createParticles(x, y, color, count = 20) {
for (let i = 0; i < count; i++) {
particles.push({
x: x,
y: y,
vx: (Math.random() - 0.5) * 8,
vy: (Math.random() - 0.5) * 8,
life: 1,
decay: Math.random() * 0.02 + 0.01,
size: Math.random() * 4 + 2,
color: color
});
}
}
// Shooting
function shoot() {
if (gameState.gameOver) return;
// Cooldown check
const now = gameState.frame;
if (now - player.lastShot < 10) return;
player.lastShot = now;
bullets.push({
x: player.x,
y: player.y - 20,
width: 4,
height: 15,
speed: 10,
color: '#ffff00'
});
createParticles(player.x, player.y - 20, '#ffff00', 5);
}
// Spawn enemies
function spawnEnemy() {
const types = [
{ width: 30, height: 30, speed: 2, color: '#ff0000', points: 10 },
{ width: 40, height: 40, speed: 1.5, color: '#ff6600', points: 20 },
{ width: 25, height: 25, speed: 3, color: '#ff00ff', points: 15 }
];
const type = types[Math.floor(Math.random() * types.length)];
enemies.push({
x: Math.random() * (canvas.width - type.width - 40) + type.width / 2 + 20,
y: -type.height,
...type,
wobble: Math.random() * Math.PI * 2
});
}
// Spawn power-ups
function spawnPowerUp() {
if (Math.random() < 0.3) {
powerUps.push({
x: Math.random() * (canvas.width - 40) + 20,
y: -20,
width: 20,
height: 20,
speed: 2,
type: 'life',
rotation: 0
});
}
}
// Collision detection
function checkCollision(obj1, obj2) {
const obj1HalfWidth = (obj1.width || 0) / 2;
const obj1HalfHeight = (obj1.height || 0) / 2;
const obj2HalfWidth = (obj2.width || 0) / 2;
const obj2HalfHeight = (obj2.height || 0) / 2;
return Math.abs(obj1.x - obj2.x) < obj1HalfWidth + obj2HalfWidth &&
Math.abs(obj1.y - obj2.y) < obj1HalfHeight + obj2HalfHeight;
}
// Update mode indicator
function updateModeIndicator() {
const indicator = document.getElementById('modeIndicator');
if (gameState.mode === 'ai') {
indicator.textContent = '๐ค AI PLAYING';
indicator.classList.add('ai-active');
} else if (gameState.mode === 'human') {
indicator.textContent = '๐ค HUMAN PLAYING';
indicator.classList.remove('ai-active');
} else {
indicator.textContent = 'IDLE';
indicator.classList.remove('ai-active');
}
}
// Update game
function update() {
if (gameState.gameOver || gameState.paused) return;
gameState.frame++;
// AI decision making - make decisions every 3 frames
if (gameState.mode === 'ai' && gameState.frame % 3 === 0) {
const state = agent.getState();
const action = agent.chooseAction(state);
// Learn from previous action
if (agent.lastState !== null) {
const reward = agent.calculateReward();
agent.learn(agent.lastState, agent.lastAction, reward, state);
}
agent.lastState = state;
agent.lastAction = action;
agent.executeAction(action);
}
// Human controls
if (gameState.mode === 'human') {
if (keys['ArrowLeft'] && player.x > player.width / 2 + 10) {
player.x -= player.speed;
}
if (keys['ArrowRight'] && player.x < canvas.width - player.width / 2 - 10) {
player.x += player.speed;
}
}
// Update player trail
player.trail.push({ x: player.x, y: player.y, life: 1 });
if (player.trail.length > 10) player.trail.shift();
player.trail.forEach(t => t.life -= 0.1);
// Update bullets
bullets = bullets.filter(bullet => {
bullet.y -= bullet.speed;
return bullet.y > -bullet.height;
});
// Update enemies
enemies.forEach(enemy => {
enemy.y += enemy.speed;
enemy.wobble += 0.05;
enemy.x += Math.sin(enemy.wobble) * 0.5;
});
// Spawn enemies
if (gameState.frame % 50 === 0) {
spawnEnemy();
}
// Spawn power-ups
if (gameState.frame % 400 === 0) {
spawnPowerUp();
}
// Update power-ups
powerUps.forEach(powerUp => {
powerUp.y += powerUp.speed;
powerUp.rotation += 0.05;
});
// Check bullet-enemy collisions
for (let bIndex = bullets.length - 1; bIndex >= 0; bIndex--) {
const bullet = bullets[bIndex];
for (let eIndex = enemies.length - 1; eIndex >= 0; eIndex--) {
const enemy = enemies[eIndex];
if (checkCollision(bullet, enemy)) {
gameState.score += enemy.points;
if (gameState.score > gameState.highScore) {
gameState.highScore = gameState.score;
}
createParticles(enemy.x, enemy.y, enemy.color, 30);
enemies.splice(eIndex, 1);
bullets.splice(bIndex, 1);
break;
}
}
}
// Check player-enemy collisions
for (let index = enemies.length - 1; index >= 0; index--) {
const enemy = enemies[index];
if (checkCollision(player, enemy)) {
gameState.lives--;
createParticles(enemy.x, enemy.y, '#ffffff', 40);
enemies.splice(index, 1);
if (gameState.lives <= 0) {
endGame();
}
}
}
// Check player-powerup collisions
for (let index = powerUps.length - 1; index >= 0; index--) {
const powerUp = powerUps[index];
if (checkCollision(player, powerUp)) {
gameState.lives++;
createParticles(powerUp.x, powerUp.y, '#00ff00', 20);
powerUps.splice(index, 1);
}
}
// Remove off-screen enemies and power-ups
enemies = enemies.filter(enemy => enemy.y < canvas.height + 50);
powerUps = powerUps.filter(powerUp => powerUp.y < canvas.height + 50);
// Update particles
particles = particles.filter(p => {
p.x += p.vx;
p.y += p.vy;
p.life -= p.decay;
p.vx *= 0.98;
p.vy *= 0.98;
return p.life > 0;
});
// Update stars
stars.forEach(star => {
star.y += star.speed;
if (star.y > canvas.height) {
star.y = 0;
star.x = Math.random() * canvas.width;
}
});
updateUI();
}
// Render game
function render() {
// Clear canvas
ctx.fillStyle = '#000';
ctx.fillRect(0, 0, canvas.width, canvas.height);
// Draw stars
stars.forEach(star => {
ctx.fillStyle = `rgba(255, 255, 255, ${star.opacity})`;
ctx.fillRect(star.x, star.y, star.size, star.size);
});
// Draw player trail
player.trail.forEach((t, i) => {
if (t.life > 0) {
ctx.fillStyle = `rgba(0, 255, 255, ${t.life * 0.3})`;
ctx.fillRect(t.x - 15, t.y - 15, 30, 30);
}
});
// Draw player with glow
ctx.shadowBlur = 20;
ctx.shadowColor = player.color;
ctx.fillStyle = player.color;
ctx.beginPath();
ctx.moveTo(player.x, player.y - player.height / 2);
ctx.lineTo(player.x - player.width / 2, player.y + player.height / 2);
ctx.lineTo(player.x, player.y + player.height / 4);
ctx.lineTo(player.x + player.width / 2, player.y + player.height / 2);
ctx.closePath();
ctx.fill();
ctx.shadowBlur = 0;
// Draw bullets
bullets.forEach(bullet => {
ctx.shadowBlur = 15;
ctx.shadowColor = bullet.color;
ctx.fillStyle = bullet.color;
ctx.fillRect(
bullet.x - bullet.width / 2,
bullet.y - bullet.height / 2,
bullet.width,
bullet.height
);
ctx.shadowBlur = 0;
});
// Draw enemies with glow
enemies.forEach(enemy => {
ctx.shadowBlur = 15;
ctx.shadowColor = enemy.color;
ctx.fillStyle = enemy.color;
ctx.beginPath();
ctx.arc(enemy.x, enemy.y, enemy.width / 2, 0, Math.PI * 2);
ctx.fill();
// Draw enemy details
ctx.fillStyle = 'rgba(255, 255, 255, 0.3)';
ctx.beginPath();
ctx.arc(enemy.x - 5, enemy.y - 5, enemy.width / 6, 0, Math.PI * 2);
ctx.fill();
ctx.shadowBlur = 0;
});
// Draw power-ups
powerUps.forEach(powerUp => {
ctx.save();
ctx.translate(powerUp.x, powerUp.y);
ctx.rotate(powerUp.rotation);
ctx.shadowBlur = 15;
ctx.shadowColor = '#00ff00';
ctx.fillStyle = '#00ff00';
ctx.fillRect(-10, -10, 20, 20);
ctx.fillStyle = '#ffffff';
ctx.font = 'bold 16px Arial';
ctx.textAlign = 'center';
ctx.textBaseline = 'middle';
ctx.fillText('+', 0, 0);
ctx.shadowBlur = 0;
ctx.restore();
});
// Draw particles
particles.forEach(p => {
const alpha = Math.floor(p.life * 255).toString(16).padStart(2, '0');
ctx.fillStyle = p.color + alpha;
ctx.fillRect(p.x - p.size / 2, p.y - p.size / 2, p.size, p.size);
});
// Draw game over
if (gameState.gameOver && gameState.mode !== 'ai') {
ctx.fillStyle = 'rgba(0, 0, 0, 0.7)';
ctx.fillRect(0, 0, canvas.width, canvas.height);
ctx.fillStyle = '#ff0000';
ctx.font = 'bold 48px Arial';
ctx.textAlign = 'center';
ctx.shadowBlur = 20;
ctx.shadowColor = '#ff0000';
ctx.fillText('GAME OVER', canvas.width / 2, canvas.height / 2 - 30);
ctx.fillStyle = '#ffffff';
ctx.font = '24px Arial';
ctx.shadowBlur = 10;
ctx.fillText(`Final Score: ${gameState.score}`, canvas.width / 2, canvas.height / 2 + 20);
ctx.fillText('Press ENTER to play again', canvas.width / 2, canvas.height / 2 + 60);
ctx.shadowBlur = 0;
}
}
// Game loop
function gameLoop() {
update();
render();
requestAnimationFrame(gameLoop);
}
// Update UI
function updateUI() {
document.getElementById('score').textContent = gameState.score;
document.getElementById('highScore').textContent = gameState.highScore;
document.getElementById('lives').textContent = gameState.lives;
}
// End game
function endGame() {
gameState.gameOver = true;
if (gameState.mode === 'ai') {
agent.gameEnded(gameState.score);
}
}
// Start game function
function startGame(mode) {
// Reset game state
gameState.score = 0;
gameState.lives = 3;
gameState.gameOver = false; // KEY FIX: Set to false to start the game
gameState.frame = 0;
gameState.lastScore = 0;
gameState.mode = mode;
// Reset player
player.x = canvas.width / 2;
player.y = canvas.height - 80;
player.trail = [];
player.lastShot = 0;
// Clear arrays
bullets = [];
enemies = [];
particles = [];
powerUps = [];
// Reset agent state
agent.lastScore = 0;
agent.lastState = null;
agent.lastAction = null;
// Update UI
updateUI();
updateModeIndicator();
}
// Reset game (for button)
function resetGame() {
startGame(gameState.mode);
}
// Event listeners
document.addEventListener('keydown', (e) => {
keys[e.key] = true;
if (e.key === ' ' && gameState.mode === 'human' && !gameState.gameOver) {
e.preventDefault();
shoot();
}
if (e.key === 'Enter' && gameState.gameOver && gameState.mode === 'human') {
e.preventDefault();
startGame('human');
}
});
document.addEventListener('keyup', (e) => {
keys[e.key] = false;
});
document.getElementById('playBtn').addEventListener('click', () => {
startGame('human');
document.getElementById('playBtn').disabled = true;
document.getElementById('aiBtn').disabled = true;
document.getElementById('stopBtn').disabled = true;
});
document.getElementById('aiBtn').addEventListener('click', () => {
startGame('ai');
document.getElementById('playBtn').disabled = true;
document.getElementById('aiBtn').disabled = true;
document.getElementById('stopBtn').disabled = false;
});
document.getElementById('stopBtn').addEventListener('click', () => {
gameState.mode = 'idle';
gameState.gameOver = true;
updateModeIndicator();
document.getElementById('playBtn').disabled = false;
document.getElementById('aiBtn').disabled = false;
document.getElementById('stopBtn').disabled = true;
});
document.getElementById('resetBtn').addEventListener('click', () => {
if (confirm('Reset high score and AI learning data?')) {
gameState.highScore = 0;
agent.qTable.clear();
agent.epsilon = 1.0;
agent.gamesCompleted = 0;
agent.scoreHistory = [];
agent.bestScore = 0;
agent.updateUI();
updateUI();
}
});
// Initialize
updateUI();
updateModeIndicator();
agent.updateUI();
// Start game loop
gameLoop();
</script>
</body>
</html>
No comments:
Post a Comment