📘 1. Introduction to Tokens

✅ Definition

A token is the smallest meaningful unit of a program that the compiler understands.

---

📌 Example

int x = 10;

👉 Tokens:

int | x | = | 10 | ;

---

🧠 Key Terms

Term	Meaning
Token	Category (e.g., identifier, keyword)
Lexeme	Actual text (e.g., x, 10)
Pattern	Rule used to define token

---

⭐ Important (Exam)

👉 Token = <Token Name, Attribute Value>

Example:

(id, pointer to symbol table)
(num, 10)

---

🧾 2. Specification of Tokens

✅ Definition

Specification of tokens means defining tokens using rules or patterns so that the lexical analyzer can identify them.

---

🔧 How Tokens are Specified?

Tokens are specified using:

🔹 1. Regular Expressions (Very Important)

👉 A regular expression is a pattern used to describe a set of strings.

---

📌 Common Token Specifications

1. Identifier

👉 Rule:

letter (letter | digit)*

👉 Examples:

x, count, total1

---

2. Number

👉 Rule:

digit+

👉 Examples:

10, 25, 100

---

3. Keyword

👉 Predefined words:

int, if, while, return

---

4. Operators

+, -, *, /, =

---

5. Delimiters

;, (, ), {, }

---

📊 Example of Token Specification Table

Token Type	Regular Expression	Example
Identifier	letter (letter	digit)*	var1
Number	digit+	123
Operator	+	-	*
Whitespace	space/tab/newline	—

---

⭐ Key Points (Exam Focus)

• Tokens are defined using regular expressions
• Each token has a pattern
• Keywords are fixed, identifiers are variable

---

🔍 3. Recognition of Tokens

✅ Definition

Recognition of tokens is the process of identifying tokens in the input string using the specified patterns.

---

🔧 How Tokens are Recognized?

Using:

🔹 Finite Automata (Very Important)

• DFA (Deterministic Finite Automaton)
• NFA (Non-Deterministic Finite Automaton)

---

📊 Flow of Recognition

Input String → Pattern Matching → Token Identified

---

🧠 Step-by-Step Recognition Process

1. Read input character
2. Match with token patterns
3. Form lexeme
4. Assign token type
5. Move to next token

---

📌 Example

Input:

sum = a + 5;

Recognition:

sum → identifier
=   → operator
a   → identifier
+   → operator
5   → number
;   → delimiter

---

🔄 4. Role of Finite Automata

✅ What is Finite Automaton?

A machine used to recognize patterns in input strings.

---

📊 Example: Identifier Recognition

Start → letter → (letter/digit)* → Accept

---

🌳 Simple DFA Diagram

(q0) --letter--> (q1)
(q1) --letter/digit--> (q1)

• q0 = start state
• q1 = accepting state

---

⭐ Key Point (Exam)

👉 “Regular Expressions → Finite Automata → Token Recognition”

---

⚠️ 5. Errors in Token Recognition

Types of Errors:

• Invalid symbol → @
• Invalid identifier → 1abc
• Unterminated string

---

🧪 6. Complete Example (Specification + Recognition)

Input:

int total = 25;

---

Step 1: Specification

• int → keyword
• total → identifier
• 25 → number

---

Step 2: Recognition

Lexeme	Token
int	keyword
total	identifier
=	operator
25	number
;	delimiter

---

🎯 7. Important Exam Concepts

👉 Frequently asked:

• Define token, lexeme, pattern
• What is token specification?
• Role of regular expressions
• What is token recognition?
• DFA for identifier
• Difference between specification and recognition

---

📝 8. Short Notes (Quick Revision)

Specification

• Defines tokens
• Uses regular expressions

Recognition

• Identifies tokens
• Uses DFA/NFA

Key Relation

Regular Expression → DFA → Token Recognition

---

📊 9. Final Summary Table

Aspect	Specification of Tokens	Recognition of Tokens
Definition	Defining token patterns	Identifying tokens in input
Tool Used	Regular Expressions	Finite Automata (DFA/NFA)
Purpose	Describe tokens	Detect tokens
Stage	Before scanning	During scanning
Example	digit+	Recognizing "123" as number
Output	Token rules	Actual tokens
Importance	High	Very High

---

✅ Final Conclusion

• Specification tells what tokens look like
• Recognition tells how tokens are found
• Both are essential for lexical analysis

---