Narain

Synthetic Data Hackathon - AMD

Fri, 31 Oct 2025 17:15:52 +0000

Finalist Submission Methodology & Technical Deep Dive

First and foremost, I’d like to extend my sincere gratitude to the AMD & Meta Hackathon organizers for orchestrating this intensive, high-impact competition especially for investing time in evaluating submissions, facilitating discussions, and reorganizing logistics to ensure fairness and transparency.

I also want to thank all fellow competitors for their thoughtful contributions and active engagement throughout the event particularly those who participated in Discord threads, shared early insights, or offered peer feedback. Your collaboration elevated the entire experience.

Special thanks to EDA Daniel and Sanyam for dedicating significant time to evaluate submissions, provide nuanced feedback, and help realign judging frameworks as needed.

Problem Context: Why Synthetic Data?

The core challenge of this hackathon centered around generating high-quality, domain-relevant synthetic datasets to train or fine-tune LLMs specifically for reasoning tasks involving character relationships and seating arrangement (linear and circular).

Initial approaches involved Synthetic data generation involving a assorted set of aptitude like questions from nvidia/OpenMathReasoning .

Early attempts at brute-force data generation using generic Aptitude datasets (including math, logic puzzles, and general knowledge Q&A) failed to produce usable training material because:

The semantic relationships between entities were either absent or inconsistent.
Outputs suffered from heavy repetition due to over-reliance on a single base model without diversification strategies.
Generated samples lacked contextual grounding making them unsuitable for fine-tuning models intended for relational reasoning.

This led us to abandon “scrape-and-prompt” approaches and instead design a multi-stage, relationship-aware synthetic data pipeline grounded in structured generation, guided prompting, and iterative refinement.

Methodology: Multi-Stage Question-Answer Pair Generation Pipeline

Prior to embarking on synthetic data generation, we dedicated substantial effort to curating a robust initial dataset. We recognized that a high-quality foundation maximizes training efficiency and long-term returns. Unsloth’s integration was pivotal throughout the hackathon, enabling seamless workflows without NaN errors—issues typically plaguing fp16 and quantized setups.

To produce synthetic QA pairs encoding meaningful character and plot relationships—mirroring reasoning in story-based or logic tasks—we built a streamlined pipeline. We began by defining semantic relationships among characters, such as parent-child, grandparent-grandchild, niece-nephew, and in-laws. For instance, queries like “Who is the mother of the White Rabbit?” drew on inferred or fictional roles when not explicit, forming the core of our prompt engineering.

Next, we assigned concrete names to these roles—e.g., Alice, White Rabbit, Cheshire Cat, Mad Hatter, March Hare—and framed questions around them for consistency. Prompts explicitly directed generation, e.g., “Create a multiple-choice question where the correct answer captures the relationship between [Character A] and [Character B], rooted in their story roles involving [Characters C, D, E].” This minimized hallucinations and boosted output alignment.

Finally, we applied constrained LLM inference: first supplying both question and correct answer to elicit relationship-based explanations, then reversing to generate narrative-plausible distractors from the question and context alone. This bidirectional validation enhanced quality, diversity, and noise reduction. Again using structured output generation.

One of the largely found exploit in arrangement was sorting

Thanks to the tokenisation of Language models.

A prominent vulnerability in seating arrangement tasks (linear or circular) involves alphabetical sorting of entity names. LLMs frequently err by defaulting to sorted orders when given random names, leading to incorrect inferences. We excluded such patterns from our training data to avoid propagating these flaws, as generated answers often replicated the errors.

Technical Implementation details

Model Selection & Training Setup

We experimented with several open-source LLMs known for strong reasoning capabilities:


Llama 2 70B	70B parameters
Qwen3-Next 80B MoE	80B MoE
Phi-4	Smaller, efficient
R1-based LLMs	Custom fine-tuned variants

All models were run locally or via cloud APIs with controlled temperature settings (typically 0.3–0.7) to balance creativity vs. accuracy.

Structured Output (grammar) Enforcement

One critical breakthrough was enforcing structured output formats during generation. Instead of free-form text, we required LLMs to return JSON-like structures containing:

{
"question": "What is the relationship between Alice and the Cheshire Cat?",
"answer": "Observer/Advisor",
"distractors": ["Mother", "Enemy", "Student"],
"relationship_type": "narrative_role",
"context_hint": "The Cheshire Cat provides cryptic advice but does not directly influence Alice's actions."
}

This enabled automated parsing, deduplication, and downstream integration into training pipelines.

{
 "topic": "TopicEnum.FAMILY_TREE",
 "question": "Symbolic notation: A + B → A is mother of B; A – B → A is brother of B; A % B → A is father of B; A × B → A is sister of B. Which of the following shows that P is the maternal uncle of Q?, answer=C",
 "reasoning": "P – M → P is brother of M; M + N → M is mother of N; N × Q → N is sister of Q. So, P is brother of Q’s mother → maternal uncle.",
 "options": ["A. Q – N + M × P", "B. P + S × N – Q", "C. P – M + N × Q", "D. Q – S % P"]
}

Question Generation model

// constraints

max_token_limit = 1024 timelimit for generation = 10 seconds ~100 tokens/s

Modeling Strategy

Initial Training Phase: SFT with System Prompts

We began by fine-tuning base LLMs (Llama 3 70B, Qwen3-Next 80B MoE) using system prompts similar to starter code but for faster iterations models like Qwen 4b was very benificial especially for RL finetuning.

We also added format enforcement instructions such as:

“Always include exactly four options labeled A, B, C, D. Never omit any keys. Do not use markdown or plain text only valid JSON.”

This reduced early-stage hallucinations and format violations.

GRPO Reward Functions

To overcome issues with output format and missing parts of the questions, we applied Guided Reinforcement Policy Optimization (GRPO) with custom rewards:

1.JSON Format Compliance 2.Presence of All Required Keys - Enforces inclusion oftopic,question,reasoning,options 3.Single-Character Answer Enum (A/B/C/D) -Prevents invalid answer labels like “Option C” or “C.”

These rewards were weighted during reinforcement learning phases to nudge the model toward clean, machine-readable outputs.

What Didn’t Work

Despite initial success, we encountered critical issues while infering:

Repetitive Outputs Due to Static System Prompts

Even after SFT, the model would generate near-identical questions repeatedly because the system prompt remained constant across generations.

Flawed Questions With Invalid Logic

Some generated questions had internal contradictions or unsolvable logic (e.g., “If A is brother of B and B is sister of C, then A is sister of C” false!).

(Unfinished Experimentation): Cross-Validation via Answer Model

Started with the idea of exploring using the trained Answer Generation Model as a validator:

For every generated question, we can pass it to the Answer Model and check validity:

Does it return a valid JSON?

Is the answer among A/B/C/D?

Does the reasoning match the expected logic?

This could creaties a self-correcting pipeline if the Answer Model couldn’t solve the question, the Question Model was penalized during GRPO updates.

This could have significantly improved logical consistency and reduced flawed samples.

ANSWER GENERATION MODEL

// constraints
{
"max_token_limit": 512,
"time_limit": "5 seconds (~100 tokens/sec)"
}

Modeling Strategy

Intuition Behind Design

Rather than treating this as a simple “answer extraction” task, we modeled it as a reasoning-first, answer-second process mimicking human problem-solving:

“First understand the logic. Then deduce the answer. Only then present it clearly.”

Supervised Fine-Tuning (SFT)

Trained on curated question-answer-reasoning triplets from our own dataset.

Each sample followed this schema:

{
"question": "...",
"reasoning": "... step-by-step derivation ...",
"answer": "C"
}

We had two training phases here

longer reasoning context
Shorter reasoning context

The shorter reasoning helps us in reducing the token output.

GRPO Reward Functions

Applied during iterative tuning to refine behavior beyond basic accuracy:


1.Exact JSON Pattern Match	1.0	Must match predefined schema exactly
2.Correct Answer Letter	2.0	Highest weight — getting the right choice matters most
3.Approximate JSON Pattern	0.5	Tolerates minor formatting deviations if meaning preserved
4.Presence of All Expected Keys	0.1 per key	Encourages completeness without over-penalizing

This weighting strategy prioritized correctness > structure > completeness, which aligned with our goal: build a reliable validator for the Question Generator (TODO).

Why Reasoning Matters More Than Just Answer

We found that the reasoning field is crucial for labeling correctness.

Example:

Question: “Who is P’s maternal uncle?”
Option C: “P – M + N × Q”
Reasoning: “P – M → P is brother of M; M + N → M is mother of N; N × Q → N is sister of Q. So P is brother of Q’s mother → maternal uncle.”

Without the reasoning, you can’t verify whether option C is truly correct — especially when symbolic notation varies.

Thus, we treated reasoning as a core component of ground truth, not just an explanation.

What Didn’t Work

Rewarding Low Token Usage in Reasoning

We initially tried rewarding shorter reasoning traces to encourage efficiency.

Failed because:

Often led to skipped steps or unjustified conclusions
Reduced model confidence and increased error rates - removed this reward entirely. Instead, focused on logical completeness and clarity of inference chain.

Summary

For final submissions, the question model used a base, unfine-tuned Phi-4 14B, while the answering model employed Qwen3 14B with LoRA training via SFT and GRPO. To ensure compatibility and fairness, we adhered closely to the provided submission code templates, minimizing deviations.

Llm Inference on Lambda

Fri, 03 Jan 2025 01:01:19 -0700

when it comes to deploying machine learning models, there are varied options to choose from depending on your scalability and cost requirements. A dedicated instance, for example, offers a stable environment for serving models but often falls short in scalability, making it less ideal for workloads with unpredictable traffic patterns. This is where a scalable distributed system like AWS Lambda.

What does Lambda provide

AWS Lambda offers a serverless architecture that scales automatically with demand, ensuring you only pay for the actual computation times used. For lightweight and quantized machine learning models. especially those finetuned for specific tasks, Lambda provides an efficient deployment option. with its support for up to 6 vCPUs and 10 GB of memory, it can handle smaller models effectively, sufficient to run some mobile YOLO models and llm models which are optimized for inference using GGML.

However deploying complex models like llama-cpp in a virtualized Lambda environment comes with unique challeges.

Restrictions on specialized CPU instructions (such as AVX512 and AMX) which are available on some of the servers that AWS provides but are not available for use.
If some lambda function is idle it will go to cold start state and the next time it is invoked it requires a startup time to allocate a machine and start up the server.

running machine learning models which are compute demanding require some careful configuration.

below i have provided some basic configuration of running llama cpp on lambda on aws. with this setup you should be good enough to run nearly all machine learning models on lambda once it is converted to gguf format

python inference script


from llama_cpp import Llama
def download_model_to_tmp():
pass
def initialize_model():
if model_exists_in_tmp(): download_model_to_tmp()
llm = Llama(
model_path="/tmp/lama-model.gguf",
)
return True
def lambda_handler(event, context):
if event['body']['health_check']: return initialize_model()
response = llm(
"Q: Name the planets in the solar system? A: ", # Prompt
max_tokens=32,
stop=["Q:", "\n"],
echo=True
)
return response

Code of the dockerfile which is required for the build

FROM --platform=linux/amd64 python:3.11-slim as build-image
ARG FUNCTION_DIR="/function"
RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
COPY inference.py .
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential \
cmake \
libopenblas-dev \
libgomp1 \
pkg-config \
&& rm -rf /var/lib/apt/lists/*
RUN CMAKE_ARGS="-DGGML_BLAS=ON -DGGML_BLAS_VENDOR=OpenBLAS \
-DCMAKE_CXX_FLAGS=\"-march=x86-64\" -DLLAMA_AVX512=OFF \
-DLLAMA_AVX2=OFF -DLLAMA_AVX=OFF -DLLAMA_FMA=OFF \
-DLLAMA_F16C=OFF -DLLAMA_BUILD_SERVER=1 \
-DLLAMA_CUBLAS=OFF -DGGML_NATIVE=OFF" \
pip install --target ${FUNCTION_DIR} llama-cpp-python
RUN pip install --target ${FUNCTION_DIR} --no-cache-dir boto3
RUN pip install --target ${FUNCTION_DIR} --no-cache-dir awslambdaric==2.0.7
COPY --from=public.ecr.aws/lambda/python:3.11 /var/runtime /var/runtime
COPY --from=public.ecr.aws/lambda/python:3.11 /var/lang /var/lang
COPY --from=public.ecr.aws/lambda/python:3.11 /usr/lib64 /usr/lib64
COPY --from=public.ecr.aws/lambda/python:3.11 /opt /opt
FROM --platform=linux/amd64 python:3.11-slim as runtime-image
ARG FUNCTION_DIR="/function"
WORKDIR ${FUNCTION_DIR}
RUN apt-get update \
&& apt-get -y install libopenblas-dev libgomp1 \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
COPY --from=public.ecr.aws/lambda/python:3.11 /var/runtime /var/runtime
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
CMD [ "inference.lambda_handler" ]

References

OMP parallelization

Wed, 20 Nov 2024 01:15:54 -0700

OpenMP (Open Multi-Processing) is an High level API for parallel programming in C, C++, and Fortran. It’s a widely-used, open-standard, and platform-independent library that allows developers to write parallel code that can run on multiple CPU cores, reducing the overall execution time of a program.

OpenMP does the heavy lifting in several areas:

Thread Management: OpenMP automatically manages thread creation, synchronization, and termination, freeing the developer from the complexity of thread management.
Work Distribution: OpenMP provides a way to distribute work among threads, ensuring that each thread has a portion of the work to execute.
Synchronization: OpenMP provides built-in synchronization mechanisms, such as barriers and locks, to ensure that threads access shared data safely.
Data Management: OpenMP provides a way to manage data sharing between threads, reducing the risk of data corruption and inconsistencies.

OpenMP makes it easy to experiment with parallelism by:

Incremental Parallelism: OpenMP allows developers to incrementally add parallelism to existing serial code, making it easier to test and refine parallel code.
Simple Directives: OpenMP uses simple directives (e.g., #pragma omp parallel) to indicate parallel regions, making it easy to write parallel code.
Portability: OpenMP code is portable across different platforms, including Windows, macOS, and Linux.

Comparison with POSIX Threads (pthreads)

POSIX threads (pthreads) is a low-level, Unix-based threading API that provides a way to create and manage threads. Here’s a comparison with OpenMP:

Note: this blog stands as a code first introduction of the directives and clauses that openmp provides most of the time you can get away with just using openmp for loop parallelization but when you want to get a little deeper into scheduling and having private and reduction variables this would stand as a all in place reference.

OpenMP Directives

OpenMP Clauses

Parallel Directive

The parallel directive is used to specify a block of code that should be executed in parallel by multiple threads. This directive is the foundation of OpenMP programming, and it is used to create a team of threads that will execute the code within the parallel region.

c Parallel Directive


#include <omp.h>
#include <stdio.h>
int main() {
#pragma omp parallel
{
printf("Hello from thread %d\n", omp_get_thread_num());
}
return 0;
}

Note: The omp_get_thread_num() function is used to get the thread number of the current thread.

Sections Directive

The sections directive is used to divide a block of code into multiple sections that can be executed in parallel by different threads. This directive is useful when you have multiple independent tasks that can be executed concurrently.

c Sections Directive


#include <omp.h>
#include <stdio.h>
int main() {
#pragma omp parallel sections
{
#pragma omp section
{
printf("Section 1: Hello from thread %d\n", omp_get_thread_num());
}
#pragma omp section
{
printf("Section 2: Hello from thread %d\n", omp_get_thread_num());
}
}
return 0;
}

Single Directive

The Single Directive is used to specify a block of code that should be executed by only one thread in a team. This directive is useful when you want to perform some operation that should only be done once, such as initializing a shared variable or printing a message.

#pragma omp single { // code to be executed by a single thread }

Example

c Single Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
#pragma omp parallel
{
#pragma omp single
{
x = 10;
printf("Thread %d initialized x to %d\n", omp_get_thread_num(), x);
}
printf("Thread %d sees x as %d\n", omp_get_thread_num(), x);
}
return 0;
}

Output

Thread 0 initialized x to 10
Thread 0 sees x as 10
Thread 1 sees x as 10
Thread 2 sees x as 10
Thread 3 sees x as 10

As you can see, only one thread (Thread 0) initializes the variable x to 10, and all other threads see the updated value of x.

Note

The Single Directive does not imply a barrier, so the other threads may continue executing without waiting for the single thread to finish its execution. If you need to ensure that all threads wait for the single thread to finish, you can use the #pragma omp barrier directive after the Single Directive.

Here’s an updated example with a barrier:

c Single Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
#pragma omp parallel
{
#pragma omp single
{
x = 10;
printf("Thread %d initialized x to %d\n", omp_get_thread_num(), x);
}
#pragma omp barrier
printf("Thread %d sees x as %d\n", omp_get_thread_num(), x);
}
return 0;
}

This ensures that all threads wait for the single thread to finish initializing x before they continue executing.

Master Directive

The master directive is used to specify a block of code that should be executed by the master thread only. The master thread is the thread that has a thread number of 0.

Syntax

#pragma omp master { // code to be executed by the master thread }

Example

c Master Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
#pragma omp parallel
{
#pragma omp master
{
x = 10;
printf("Master thread: x = %d\n", x);
}
printf("Thread %d: x = %d\n", omp_get_thread_num(), x);
}
return 0;
}

In this example, the master thread sets the value of x to 10 and prints a message. All threads then print the value of x.

Note

The master directive does not imply a barrier, so the other threads may continue executing without waiting for the master thread to finish its execution.

When to use

The master directive is useful when you need to perform some operation that should only be done once, such as initializing a shared variable or printing a message.

Comparison with Single Directive

The master directive is similar to the single directive, but the single directive can be executed by any thread, while the master directive is always executed by the master thread.

Here is an example that demonstrates the difference:

c Master Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
#pragma omp parallel
{
#pragma omp single
{
x = 10;
printf("Single thread: x = %d\n", x);
}
printf("Thread %d: x = %d\n", omp_get_thread_num(), x);
}
return 0;
}

In this example, the single directive is used to set the value of x to 10. The thread that executes the single directive may not be the master thread.

Critical Directive

The critical directive is used to specify a block of code that should be executed by only one thread at a time. This directive is useful when you need to protect a critical section of code that should not be executed concurrently by multiple threads.

Syntax

#pragma omp critical [(name)] { // code to be executed by only one thread at a time }

Example

c Critical Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
#pragma omp parallel
{
#pragma omp critical
{
x = x + 1;
printf("Thread %d: x = %d\n", omp_get_thread_num(), x);
}
}
return 0;
}

In this example, the critical directive is used to protect the increment of the variable x. Only one thread can execute the critical section at a time, ensuring that the increment is atomic.

Named Critical Sections

You can also specify a name for the critical section, which allows you to have multiple critical sections that can be executed concurrently.

#pragma omp critical (name) { // code to be executed by only one thread at a time }

Example

c Critical Directive


#include <omp.h>
#include <stdio.h>
int main() {
int x = 0;
int y = 0;
#pragma omp parallel
{
#pragma omp critical (x_lock)
{
x = x + 1;
printf("Thread %d: x = %d\n", omp_get_thread_num(), x);
}
#pragma omp critical (y_lock)
{
y = y + 1;
printf("Thread %d: y = %d\n", omp_get_thread_num(), y);
}
}
return 0;
}

In this example, we have two named critical sections, x_lock and y_lock. These critical sections can be executed concurrently, but only one thread can execute each critical section at a time.

Best Practices

Use the critical directive to protect critical sections of code that should not be executed concurrently by multiple threads.
Use named critical sections to allow multiple critical sections to be executed concurrently.
Minimize the use of critical sections, as they can introduce performance overhead.
Use other synchronization mechanisms, such as locks or atomic operations, when possible.

Barrier Directive

The barrier directive is used to synchronize all threads in a team. When a thread reaches a barrier, it waits until all other threads in the team have also reached the barrier. Once all threads have reached the barrier, they can all proceed past it.

Syntax

#pragma omp barrier

Example

Here’s an example demonstrating the use of the barrier directive:

c Barrier Directive


#include <omp.h>
#include <stdio.h>
#include <unistd.h>
int main() {
#pragma omp parallel num_threads(4)
{
int thread_num = omp_get_thread_num();
// First part of the work
printf("Thread %d: Starting first part of work\n", thread_num);
sleep(thread_num); // Simulate different work times
#pragma omp barrier
// Second part of the work
printf("Thread %d: Starting second part of work\n", thread_num);
}
return 0;
}

Output

Thread 0: Starting first part of work
Thread 3: Starting first part of work
Thread 2: Starting first part of work
Thread 1: Starting first part of work
Thread 0: Starting second part of work
Thread 1: Starting second part of work
Thread 2: Starting second part of work
Thread 3: Starting second part of work

Explanation

Four threads are created.
Each thread prints a message and then sleeps for a duration equal to its thread number (simulating different work times).
The barrier directive ensures that all threads wait at this point until every thread has reached the barrier.
Once all threads have reached the barrier, they all proceed to print the second message.

Important Notes

The barrier directive is implicit at the end of parallel, for, sections, and single constructs, unless a nowait clause is specified.
Using too many barriers can negatively impact performance, as threads may spend a lot of time waiting for each other.
Barriers should be used judiciously to ensure correct program behavior while minimizing synchronization overhead.
Be cautious of potential deadlocks when using barriers, especially in complex parallel regions.

The barrier directive is a powerful tool for synchronizing threads, but it should be used carefully to balance correctness and performance in your parallel programs.

Private Clause

The private clause is used to specify that each thread should have its own copy of the listed variables. This is useful when you want each thread to work with its own instance of a variable, preventing data races and ensuring thread safety.

Syntax

#pragma omp parallel private(variable1, variable2, …)

#pragma omp for private(variable1, variable2, …)

Example

Here’s an example demonstrating the use of the private clause:

c Private Clause


#include <omp.h>
#include <stdio.h>
int main() {
int i, sum = 0;
#pragma omp parallel for private(i) reduction(+:sum)
for (i = 1; i <= 100; i++) {
int temp = i * i;
sum += temp;
printf("Thread %d: i = %d, temp = %d\n", omp_get_thread_num(), i, temp);
}
printf("Sum of squares from 1 to 100: %d\n", sum);
return 0;
}

Output

The output might look something like this (note that the order of thread execution may vary):

Thread 0: i = 1, temp = 1
Thread 0: i = 2, temp = 4
...
Thread 1: i = 26, temp = 676
Thread 1: i = 27, temp = 729
...
Thread 2: i = 51, temp = 2601
Thread 2: i = 52, temp = 2704
...
Thread 3: i = 76, temp = 5776
Thread 3: i = 77, temp = 5929
...
Sum of squares from 1 to 100: 338350

Explanation

The i variable is declared as private. This means each thread gets its own copy of i.
Inside the parallel for loop, each thread works with its own i, preventing any data races.
The temp variable is implicitly private because it’s declared inside the parallel region.
Each thread calculates the square of its assigned numbers and adds them to the shared sum variable (using the reduction clause, which we’ll cover in a future topic).

Important Notes

Variables declared as private are not initialized. They contain garbage values when entering the parallel region unless explicitly initialized.
Changes made to private variables are not visible outside the parallel region.
If you need to initialize private variables or use their values after the parallel region, consider using the firstprivate or lastprivate clauses (which we’ll cover next).
Array sections can also be made private using the syntax private(array[start:length]).
The private clause is often used with loop index variables to ensure each thread has its own copy of the loop counter.

The private clause is a fundamental tool in OpenMP for managing data in parallel regions, helping to prevent data races and ensure correct parallel execution.

Firstprivate Clause

The firstprivate clause is similar to the private clause, but it also initializes the private copies with the value of the original variable before entering the parallel region.

Syntax

#pragma omp parallel firstprivate(variable1, variable2, …)

#pragma omp for firstprivate(variable1, variable2, …)

Example

Here’s an example demonstrating the use of the firstprivate clause:

c Private Clause


#include <omp.h>
#include <stdio.h>
int main() {
int x = 5;
#pragma omp parallel firstprivate(x)
{
printf("Thread %d: x initially = %d\n", omp_get_thread_num(), x);
x += omp_get_thread_num();
printf("Thread %d: x after modification = %d\n", omp_get_thread_num(), x);
}
printf("Outside parallel region: x = %d\n", x);
return 0;
}

Output

The output might look something like this:

Thread 0: x initially = 5
Thread 1: x initially = 5
Thread 2: x initially = 5
Thread 3: x initially = 5
Thread 0: x after modification = 5
Thread 1: x after modification = 6
Thread 2: x after modification = 7
Thread 3: x after modification = 8
Outside parallel region: x = 5

Explanation

The variable x is initialized to 5 before the parallel region.
Using firstprivate(x), each thread gets its own copy of x, initialized to 5.
Each thread then modifies its private copy of x by adding its thread number.
After the parallel region, the original x remains unchanged (still 5).

Important Notes

firstprivate is particularly useful when you need to use the initial value of a variable in a parallel region, but also want each thread to have its own copy.
For complex objects (like C++ objects with constructors), firstprivate ensures that the copy constructor is called to initialize the private copies.
firstprivate can have a performance overhead compared to private, especially for large objects, as it needs to initialize each private copy.
Changes made to firstprivate variables are not visible outside the parallel region unless you use other mechanisms (like reduction or lastprivate).
You can use firstprivate with arrays, but be cautious with large arrays as it will create a full copy for each thread.

Comparison with private

private creates uninitialized copies for each thread.
firstprivate creates initialized copies, with the value from before entering the parallel region.

The firstprivate clause is very useful when you need to work with initialized private copies in your parallel regions, combining the benefits of data isolation with proper initialization.

Lastprivate Clause

The lastprivate clause is used to specify that the value of one or more private variables should be copied back to the original variables after the parallel region or loop. Specifically, it copies the value from the last logical iteration of the loop or the last section of a sections construct.

Syntax

#pragma omp parallel for lastprivate(variable1, variable2, …)

#pragma omp sections lastprivate(variable1, variable2, …)

Example

Here’s an example demonstrating the use of the lastprivate clause:

c Private Clause


#include <omp.h>
#include <stdio.h>
int main() {
int i, x = 0;
#pragma omp parallel for lastprivate(x)
for (i = 0; i < 100; i++) {
x = i;
printf("Thread %d: i = %d, x = %d\n", omp_get_thread_num(), i, x);
}
printf("After parallel for: x = %d\n", x);
return 0;
}

Output

The output might look something like this (note that the order of thread execution may vary):

Thread 0: i = 0, x = 0
Thread 1: i = 25, x = 25
Thread 2: i = 50, x = 50
Thread 3: i = 75, x = 75
...
Thread 3: i = 99, x = 99
After parallel for: x = 99

Explanation

The variable x is declared as lastprivate.
Each thread has its own private copy of x during the parallel for loop.
After the loop completes, the value of x from the last logical iteration (i = 99) is copied back to the original x.
Outside the parallel region, x has the value from the last iteration (99).

Important Notes

lastprivate is particularly useful when you need the final value of a variable after a parallel loop or sections construct.
For loops, the “last” iteration is determined by the logical order of the loop, not the actual last thread to finish.
For sections constructs, the last section in lexical order determines the lastprivate value.
lastprivate can be combined with firstprivate if you need both initialization and final value preservation.
There’s a potential performance overhead with lastprivate, as it requires additional synchronization to determine and copy the final value.

Comparison with private and firstprivate

private: Creates uninitialized copies, final values are not preserved.
firstprivate: Creates initialized copies, final values are not preserved.
lastprivate: May create uninitialized copies (unless combined with firstprivate), but preserves the final value from the logically last iteration or section.

The lastprivate clause is valuable when you need to capture the final state of a variable after parallel execution, especially in loops where the final iteration produces an important result.

Reduction Clause

The reduction clause is used to perform a reduction operation on one or more variables across all threads in a parallel region. It’s particularly useful for operations like summing, finding the maximum or minimum, or performing logical operations across a set of values computed in parallel.

Syntax

#pragma omp parallel for reduction(operator:variable)

Where operator can be:

+, -, *, &, |, ^, &&, || for built-in types
max, min for arithmetic types

Example

Here’s an example demonstrating the use of the reduction clause to calculate the sum of squares:

c Private Clause


#include <omp.h>
#include <stdio.h>
int main() {
long long sum = 0;
int i, n = 1000000;
#pragma omp parallel for reduction(+:sum)
for (i = 1; i <= n; i++) {
sum += i * i;
}
printf("Sum of squares from 1 to %d: %lld\n", n, sum);
return 0;
}

Output

The output will be:

Sum of squares from 1 to 1000000: 333333833333500000

Explanation

The sum variable is specified as a reduction variable with the + operator.
Each thread computes a partial sum for its chunk of the loop.
OpenMP automatically combines these partial sums using the specified operator (+ in this case).
The final result is stored in the original sum variable.

Important Notes

OpenMP creates a private copy of the reduction variable for each thread, initialized according to the reduction operator (e.g., 0 for +, 1 for *, etc.).
At the end of the parallel region, these private copies are combined using the specified operator to produce the final result.
Reduction operations are both thread-safe and efficient, as they avoid the need for explicit synchronization.

You can specify multiple reduction variables in one clause:

#pragma omp parallel for reduction(+:sum1,sum2) reduction(max:maxval)

Custom reduction operations can be defined for user-defined types in C++ using the declare reduction directive.

Common Use Cases

Summing values: reduction(+:sum)
Finding maximum: reduction(max:max_value)
Finding minimum: reduction(min:min_value)
Logical AND of conditions: reduction(&&:all_true)
Logical OR of conditions: reduction(||:any_true)

Performance Considerations

Reductions are generally very efficient, as they allow each thread to work independently and combine results only at the end. This often leads to good scalability. However, for very small loops or when the reduction operation is very simple, the overhead of creating and managing threads might outweigh the benefits of parallelization.

The reduction clause is a powerful feature in OpenMP that simplifies parallel aggregation operations and helps avoid common pitfalls like race conditions when accumulating results across threads.

Schedule Clause

The schedule clause is used to specify how iterations of a loop are divided among threads in a parallel region. It allows you to control the workload distribution, which can significantly impact the performance of your parallel program.

Syntax

#pragma omp parallel for schedule(kind[, chunk_size])

Where kind can be:

static
dynamic
guided
auto
runtime

Types of Schedules

Static Schedule
- Iterations are divided into chunks of size chunk_size and distributed to threads in a round-robin fashion.
- If chunk_size is not specified, iterations are divided equally among threads.
Dynamic Schedule
- Iterations are divided into chunks of size chunk_size.
- Each thread requests a new chunk when it finishes its current chunk.
Guided Schedule
- Similar to dynamic, but the chunk size starts large and decreases over time.
- The chunk size is proportional to the number of unassigned iterations divided by the number of threads.
Auto Schedule
- The decision of scheduling is delegated to the compiler and/or runtime system.
Runtime Schedule
- The schedule is determined at runtime by the OMP_SCHEDULE environment variable.

Example

Here’s an example demonstrating different schedule types:

c Private Clause


#include <omp.h>
#include <stdio.h>
void test_schedule(const char* schedule_type) {
int i, n = 20;
#pragma omp parallel for schedule(runtime)
for (i = 0; i < n; i++) {
printf("Thread %d executing iteration %d\n", omp_get_thread_num(), i);
}
printf("\nTested with schedule type: %s\n\n", schedule_type);
}
int main() {
const char* schedules[] = {"static", "static,2", "dynamic", "dynamic,2", "guided", "auto"};
int num_schedules = sizeof(schedules) / sizeof(schedules[0]);
for (int i = 0; i < num_schedules; i++) {
omp_set_schedule(omp_sched_runtime, 0);
setenv("OMP_SCHEDULE", schedules[i], 1);
test_schedule(schedules[i]);
}
return 0;
}

Choosing the Right Schedule

Use static when iterations have similar workloads and you want to minimize scheduling overhead.
Use dynamic or guided when iterations have varying workloads to achieve better load balancing.
Use auto when you’re unsure and want the system to decide.
Use runtime for flexibility in testing different schedules without recompiling.

The choice of schedule can significantly affect performance, especially for loops with irregular workloads or when dealing with NUMA (Non-Uniform Memory Access) architectures.

Understanding floating point numbers

Fri, 21 Jun 2024 16:24:31 -0700

introduction

In 1991, a Patriot missile defense system in Saudi Arabia failed to intercept an incoming Scud missile, resulting in the tragic deaths of 28 American soldiers. The cause? A tiny floating point error that accumulated over time, throwing off the system’s timing by just 0.34 seconds. This incident highlights the critical importance of understanding floating point numbers in computer systems. Floating point representation allows computers to work with a wide range of real numbers, from the microscopic to the astronomical. However, it also introduces complexities and potential inaccuracies that programmers must carefully navigate.

It is important to understand the way we represent floating point numbers in computers to avoid unwanted edge cases in programming. if you are considering to use float32 for managing your finances just because you would be able to represent cents be mindful that IEEE 754 skips a number after 2^24 (16,777,217). At this point you are not only missing numbers but errors due to calculations keeps on adding up.

Disclaimer Throughout this writeup i will be consistently switching between hardware and its usecases in
writing applications as we are bound by the hardware design

how is floating point numbers represented in binary

Binary Representations

sing base-2 (binary) representation for floating point numbers minimizes rounding errors compared to higher bases. Higher base systems often introduce unused bits, leading to inefficient memory usage. Furthermore, binary representation aligns with how electronic systems naturally process and store data, making it the optimal choice for computational tasks

Real numbers are infinite between any two integers, but representing them on computers with fixed bit widths involves trade-offs. The density of floating point numbers decays exponentially as we move away from zero, meaning that more precision is available near zero and less as values grow larger. Squeezing infinitely many real numbers into a fixed bit width necessitates approximations. As a result, there are more representable numbers between -2 and 2 than in other regions of the number line.

c floating point numbers skipping


#include <stdio.h>
#include <math.h>
int main() {
float a = (float)pow(2, 24);
for (int i = 1; i < 10; i++) {
printf("%d, %f\n", i, a + i);
}
}
// 2**24 + 1 = 16777216.000000
// 2**24 + 2 = 16777218.000000
// 2**24 + 3 = 16777220.000000
// 2**24 + 4 = 16777220.000000
// 2**24 + 5 = 16777220.000000
// 2**24 + 6 = 16777222.000000
// 2**24 + 7 = 16777224.000000
// 2**24 + 8 = 16777224.000000
// 2**24 + 9 = 16777224.000000

From the above example we observe that as numbers get larger the spacing between consecutive representable numbers increases. The above case at 2^24 floating point format cannot represent every integer precisely beyond this point.

2^24 + 1 the floating point precision is insufficient to capture the difference of just 1 due to rounding error
2^24 + 2 the floating point precision allows 16777218 to be represented and skips over 16777217.

this phenomenon is called as floating point precision loss at higher magnitudes, more integers are skipped because the system can’t represent every possible number within the limited floating point precision.

FACT:
javascript uses float64 to represent all numeric data types

History of Floating point numbers

Initially, computers only supported integer arithmetic, as it was sufficient for early applications. However, as computational needs grew, so did the necessity to represent and manipulate decimal numbers. To address this, programmers began emulating floating-point arithmetic through complex algorithms, which required more operations than simple integer addition or subtraction.

Float Emulator

Recognizing the inefficiency of these emulations, chip manufacturers started designing dedicated hardware, such as floating-point registers, to execute floating-point instructions in a single cycle. These registers adhere to the IEEE 754 standard, which defines the floating-point format with 1 sign bit, 23 mantissa bits, and 8 exponent bits. Sticking to this standard is critical for ensuring consistency across platforms, allowing programmers to achieve identical results on different systems—vital for reliable and predictable computation.

Evolution

With the rising concern of the memory wall chip manufacturers build custom low precision floating point arithmatic such as float8, float16, bfloat16 and tensor-float32.

where float32 is the standard representation

Efficiency to high precision calculations

Floating-point operations, due to the limitations of their representation, come with some inherent drawbacks. One key issue is that they are not strictly commutative—meaning a+b≠b+a in certain cases, particularly in low-precision arithmetic. This non-commutativity increases rounding errors, especially in embarrassingly parallel applications, where threads process operands out of order. While floating-point arithmetic typically works well in serial execution, issues arise when code is parallelized or when the order of operations (like in loops) is changed, leading to slight variations in results.

Because of these precision issues, it is a common practice not to validate floating-point results by checking for exact equality. Instead, computations are often considered correct if the results fall within a specified tolerance range, ensuring that small rounding errors do not invalidate the outcome.

Further reads

Distributed Training

Sun, 07 Apr 2024 12:44:46 -0700

Training advanced neural networks, particularly deep learning models such as transformers, presents significant computational challenges. These sophisticated models require substantial computational power, often exceeding the capabilities of the single GPU machine. The primary hurdles stem from the immense number of parameters that need to be calculated and updated during the training process. This not only demands extensive GPU memory but also nencassitates the use of large batch sizes to achieve efficient learning and convergence.

As the complexity of models grows, So does the need for more advanced hardware accelerators. GPUs (Graphics Processing Units) have become essential in the field of deep learning due to their ability to handle parallel computations effectively. These accelerators significantly reduce training time by distributing the computational load across multiple cores. GPUs whith their high throughput, are particularly well-suited for tasks involving large batch sizes and extensive data parallelism. These accelerators are invaluable for training extremely large models that would otherwise be infeasible on traditional hardware.

The necassity for such powerful accelerators highlights the ever growing demands of deep learning technologies and sets the stage for exploring distributed training methods, which aim to further amplify the capabilities of induvidual training sessions by leveraging multiple GPUs or TPUs simultaneosly.

Essentials of distributed Training

Lets have a refresher on how deep learning models work. Deep Learning models operate on a mechanism known as gradient descent, a fundamental technique used to optimize these models by mimicking a data representations. During the training process, a model adjusts its internal parameters or weights based on the difference between its predictions and the actual outputs. This difference is qualified through a loss function, which measures the model performance. The backward pass of training involves updating these weights by calculating gradients, which are essentially directions and magnitudes for improving model predictions.

conventional deep learning flow

To manage these gradients, optimizers like Adam are employed. These optimizers not only update the weights based on the gradients but also keep track of past gradient values to make informed adjustments, enhancing the efficiency and stability of the learning process.

In the context of distributed deep learning, the training process is expanded across multiple machines. This setup requires a synchronized forward pass, where each device uses the same model weights but different subsets of input data. This approach enhances the model’s ability to generalize across diverse data sets and helps mitigate the influence of outliers.

After the forward pass, gradients calculated on different machines are collected and aggregated. This aggregation is crucial it ensures that all partial insights from the distributed datasets are combined to update the model in a comprehensive manner. Once the weights are updated, they are redistributed to all devices to maintain consistency across the model for subsequent training iterations.

This distributed approach not only speeds up the training process by parallelizing tasks but also allows for handlinf larger models and datasets that would be impractical to process on a single machine.

Scaling Up with distributed techniques

The need to train increasingle large models, some with billions of parameters, has necassitated a shift towards distributed training across multiple GPUs. This shift is driven by the fact that many advanced models cannot fit into memory of a single GPU. Instead, distributing the computational workload across sevaral GPUs offers a feasible solution. The practive of distributed training gained traction when CNNs were being trained on large datasets like ImageNet. In such networks, the initial convolutional layers are particularly computation-intensive due to the stencil operation required to process the entire image. This realization led to early efforts in distributed training focusing primarily on dividing the compute heavy tasks among multiple GPUs. This kind of splitting model parameters across multiple GPUs is called model parallelism.

parallelism in deep learning perspective is of the following types:

model parallel
data parallel
tensor parallel

Model Parallel

model parallel is the easiest to implement as we dont have multiple replicas of the same parameter and no reduction is required. The only change required from the normal training routine is at some point the intermediate outputs need to be transferred from one GPU to the other.

Data Parallelism

data parallelism is a cornerstone of distributed training strategies, especially when dealing with extensive deep learning models. In this approach, each GPU holds a complete copy of the model, ensuring that every unit can independently process different subsets of the overall dataset. While this method effectively utilizes the parallel nature of GPUs, it introduces significant complexity during the backward pass, particularly in gradient accumulation and synchronization.

The challenge arises from the need to integrate gradients computed induvidually on each GPU. Since different GPUs process different data, the generate unique gradients, which must be averaged before updating the model’s weights. This averaging is crucial because it ensures that the model learns uniformly from the entire dataset rather than just fragments of it.

There was a trick used to train models of larger batch size than possible using gradient accumlation, this method computes gradients every batch but only updates the weights interleaved.

Several methods are employed to manage this averaging process across machines. A commonly used technique involves a parameter server architecture. In this setup, a central server aggregates the gradients from all GPU’s. This method, however has scalability limitations. As the number of GPUs increases often going beyond ten the parameter server can become a bottleneck, straining under the heavy load of receiving and processing gradients from each unit. This can potentially halt the training process, affecting efficiency and scalability.

Parameter server

Frameworks like tensorflow have addresed some of these issues by implementing advanced versions of the parameter server model, which are designed to handle large clussters more effectively. In addition to the challenges of gradient averaging, data parallelism must also manage synchronization issues that arise when nodes vary in computational speed. This variation can introduce a problem known as staleness, where the updates from slower nodes lag behind those from faster ones.

During a distributed training session, each GPU (or node) processes a different batch of data and computes gradients independently. Ideally, these gradients are synchronized across all nodes to update the model weights uniformly. However, if one node takes significantly longer to process its batch, it causes a delay in the aggregation of gradients. As a result, faster nodes must wait for the slower ones to finish computation, leading to idle time that can significantly reduce overall system efficiency.

There is additional overhead when dealing with fault tolerency in these synchronization, but that is beyond the scope of this discussion.

This staleness not only affect the synchronization step but can also impact the accuracy and convergence of the model. If gradients from slower nodes are outdated by the time they are integrated, they might not accurately represent the current state of the model leading to suboptimal updates.

Model Synchronization

To mitigate these issues, various synchronization strategies are employed. One common approach is to use barrier synchronization, where all nodes must reach a certain point in the computation before any of them can proceed. This ensures that all gradient updates are based on the same version of the model, preventing staleness. However this method can lead to increased waiting times, as all nodes are forced to operate at the pace of the slowest one.

Another approach is to implement asynchronous updates, where nodes update the central model with their gradients as soon as they are ready, without waiting for the slowest nodes. This mothod can improve the efficiency of the system by reducing idle time but at the cost of introducing more staleness into model updates, which can complicate convergence.

Despite These bottleneck issue can still pose significant challenges, prompting the development and adoption of more sophisticated techniques such as ring-allreduce. This method distributes the task of gradient aggregation across all devices, therby eliminating the need for a central parameter server and significantly reducing the communication overhead.

Horovod

Horovod is a distributed training framework introduced by Uber by modifying the common ring-reduce algorithm this significantly enhances the efficiency of data synchronization across multiple GPUs by employing the ring allreduce algorithm. This method is specifically designed to minimize the communication overhead by reducing the number of simultaneous data transfers, thus avoiding bandwidth saturation and ensuring a stable and fast data exchange.

ring all-reduce

The ring-allreduce algorithm operates by organizing each GPU into a virtual ring topology. It begins by dividing the gradient data into equal segments and assigning each segment to a GPU. In this setup, each node is responsible for sending a segment of its data to its neighbor on the right and simultaneously recieving a segment from its neighbor on the left. Upon recieving a segment, a node quickly combines it with its correspponding segment a process known as reduction. After reducing the data, the node passes the resultant data to the next node in the ring. This process of rotating and reducing continues until every segment has circulated back to its original node.

By the end of this cyclem each GPU has recieved segments of data from all other GPUs, ensuring that each segment is fully reduced set of gradients from across the network. These averaged gradients are then used to update the model weights on each GPU, synchronizing the model state across all nodes to maintain uniformly for the next training iteration.

One of the primary benifits of horovod is its efficient use of network resources. The framework manages to distribute the communication load evenly across all participating GPUs. By transferring the weights in a round-robin fashion and processing the data in manageble parts, Horovod stays within the available bandwidth limits. This careful management helps in reducing network congestion and ensures a smoother gradient averaging process making horovod particularly advantageous in environments where network resources are constraint.

Exploring PyTorch’s Approach to Distributed training

Pytorch offers a robust distributed training framework that differs notably from Horovod’s methodology. Unline Horovod, which synchronizes gradients after they are computed across all GPUs Pytorch synchronizes gradients as soon as they become available, facilitating a more dynamic and efficient workflow. This immediate synchronization is part of PyTorch’s Distributed Data Parallel (DDP) architecture, which is designed to enhance performance, particularly in environments where bandwidth might be a limiting factor [it always is].

The key to Pytorch’s efficiency lies in its ability to overlap gradient computation with gradient synchronization. Here’s how this process typically unfolds: pytorch groups gradients into what are called buckets. as soon as one of these buckets are filled with computed gradients, it doesn’t wait for the rest of the model to finish computing its gradients. Instead it immediately begins the synchronization process for that bucket. This method allows DDP to initialize the communication phase much earlier than if it had to wait for the entire backward pass to complete.

This overlapping of computation and communication is highly benificial. It ensures that the GPUs are not left idle waiting for data, thus making full use of the network bandwidth and reducing overall training time. By starting the synchronization process as a portion of the gradients is ready, pytorch minimizes the downtime that each GPU might otherwise experience if it had to wait for the entire network’s gradients to be computed befire beginning synchronization.

Pytorch provides two common methods are used to facilitate the training of deep learning models across multiple GPUs: Data Parallel (DP) and Distributed Data Parallel (DDP). While both approaches aim to distribute the training workload across several computing units, They are fundamentally different in how they manage and execute this distribution.

Distributed Data Parallel (DDP):

Distrbuted Data Parallel, is optimized for both single machine and multi machine configurations. DDP improves upon the basic idea od DP by reducing the communication bottleneck observed in DP. It does this by eliminating the need for a master GPU. Instead, each GPU independently computes the gradients from its subset of the data, and then a collective communication operation (like all-reduce) is employed to average these gradients across all devices without funneling through a single point. DDP uses processes instead of theads for handling parallel tasks across multiple GPUs. This design choice is pivotal for enhancing performance and avoiding common pitfalls associated with multithreading in python, such as Global Interpreter Lock (GIL) constraints. Additionally, processes improve scalability across varied hardware and network architectures. They support robust and reliable communication mechanisms, such as NCCL, MPI. These are specifically optimized for inter process communication, enhancing the overall efficiency and effictiveness of distributed training operations.

Some Key words of DDP are:

WORLD_SIZE: defines the total number of workers n1 nodes n2 gpus = n1 * n2
WORLD_RANK: identifies the worker ranges from 0..(n1*n2)
LOCAL_RANK: defines the id of a worker within a node or indirectly refers to the gpu rank

References

About

Fri, 05 Apr 2024 21:11:49 -0700

Hello! I’m Narain. I tinker with machine learning models for a living, diving into the nuts and bolts of deep learning, GPU acceleration, and cloud infrastructures. When I’m not busy breaking stuff, I might be pushing the boundaries of what should (or shouldn’t) be done in C++.

Beyond the screen, I’m a karate instructor and an open-source enthusiast. Whether it’s enhancing AI frameworks or tinkering efficient code, I’m all about making a difference. When I’m looking to unwind, I enjoy exploring culinary arts and capturing life’s moments through my lens—usually involving scenes from nature or candid shots of daily life.

Distributed_dl

Thu, 21 Mar 2024 16:24:23 -0700

Distributed Deep learning training

Training routine

Data Parallel

model parallelism

pipeline parallelism

Tensor parallelism

Deepspeed

Deepspeed zero

Online Softmax

Mon, 26 Feb 2024 16:32:46 -0700

One of the most important task in deep learning is classification. This involves predicting the class to which a given input data belongs. Models such as convolution neural networks (CNN) and Large language models use classification layers. These models produce output predictions for all possible classes, but these predictions are not immediately usable as they can be any floating point number. The softmax function is essential in converting these raw model outputs, known as logits, into probabilities that sum to one, making them interpretable

The role of Softmax in Classification

Why softmax?

Raw model outputs, or logits can lie anywhere within the floating-point number range, making them difficult to interpret. The Softmax function addresses this by normalizing the logits into a probability distribution. This allows us to understand the model’s confidence in its predictions.

Steps to calculate Sofmax

Simple Softmax implementation

Normalizing by maximum logit:
- The first step involves calculating the maximum value among the model’s outputs. This helps in normalizing the outputs and prevents potential overflow issues during exponentiation.
Exponentiation:
- Next we subtract the maximum logit from each logit and calculate the exponent of these normalized values. This step transforms the logits into a form where they can be compared proportionally.
Division by sum of exponents:
- Finally we divide each exponentiated value by the sum of all exponentiated values. This step ensures that the resulting probabilities sum to one, providing a valid probability distribution.

python Softmax implementation pytorch


import torch
def softmax(x):
"""Compute softmax values for each set of scores in x."""
if not isinstance(x, torch.Tensor):
x = torch.tensor(x, dtype=torch.float32)
x_max = x.max(dim=-1, keepdim=True).values
e_x = torch.exp(x - x_max)
return e_x / e_x.sum(dim=-1, keepdim=True)
# Example usage
input_tensor = torch.tensor([1.0, 2.0, 3.0])
softmax_output = softmax(input_tensor)
print(softmax_output)

As we can see, the standard Softmax algorithm has some drawbacks. One significant issue is that it requires multiple passes over the entire tensor to calculate the softmax, makind it less cache-friendly and potentially inefficient.

Introducing Online Softmax

To address these inefficiencies, we can explore an alternative approach known as online softmax. This method aims to improve computational efficiency and cache performance by processing the data in a more streamlined manner.

Online Softmax

Single pass for maximum and sum calculation:
- Instead of first finding the maximum value in one pass and then computing the sum of exponentials in another, this code combines these operation. When a new maximum value is found the sum is adjusted accordingly, ensuring accuracy without additional passes.
Numerical Stability
- By subtracting the maximum value maxval from each element before exponentiation, the code prevents potential overflow issues that could occur with large input values.
Efficiency:
- This approach is more cache-friendly as it reduces the number of passes over the data, thus maximising the number of times the data needs to be fetched from memory.

C Online Softmax implementation


{#include <math.h>
#include <float.h> // For FLT_MAX
void softmax_online(const float *input, float *output, int n) {
float max_val = -INFINITY;
float sum = 0.0f;
for (int i = 0; i < n; i++) {
float maxval_prev = max_val;
if (inp_row[j] > max_val) {
max_val = inp_row[j];
sum = sum + expf(maxval_prev - max_val) + expf(inp_row[j] - max_val);
} else {
sum += expf(inp_row[j] - max_val);
}
}
for (int i=0; i<n; i++) {
output[i] = expf(inp_row[i] - max_val) / sum;
}
}

C Online Softmax implementation CUDA


template <typename T>
__global__
void softmax_kernel_v2(T* qk_buf_, /*const T* attr_mask, */const int batch_size, const int head_num,
const int seq_len, const T scaler)
{
// int batch_id = blockIdx.x / head_num / seq_len;
// int seq_id = blockIdx.x % seq_len;
int qk_offset = blockIdx.x * seq_len;
// int mask_offset = batch_id * seq_len * seq_len + seq_id * seq_len;
__shared__ float s_sum, s_max;
float qk = threadIdx.x < seq_len ? (float)qk_buf_[threadIdx.x + qk_offset] : 0.0f;
// float mask_val = threadIdx.x < seq_len ? (float)attr_mask[threadIdx.x + mask_offset] : 0.0f;
// mask_val = (1.0f - mask_val) * -10000.0f;
// float tmp = threadIdx.x < seq_len ? (float)(qk * (float)scaler + mask_val) : -1e20f;
float tmp = -1e20f;
float max_val = blockReduceMax<float>(tmp);
if(threadIdx.x == 0)
s_max = max_val;
__syncthreads();
float qk_tmp = threadIdx.x < seq_len ? __expf((float)(tmp - s_max)) : 0.0f;
float sum_val = blockReduceSum<float>(qk_tmp);
if(threadIdx.x == 0)
{
s_sum = sum_val + 1e-6f;
}
__syncthreads();
if(threadIdx.x < seq_len)
qk_buf_[threadIdx.x + qk_offset] = (T)(qk_tmp / s_sum);
}

This code becomes even more efficient when used on GPU’s as all tensor data can be moved to GPU’s shared memory

References

Online normalizer calculation for softmax Paper

Sentencepiece Tokenizer

Mon, 15 Jan 2024 04:56:12 -0700

Sentencepiece tokenizer

Introduction

In the rapidly evolving field of natural language processing (NLP), the efficiency and accuracy of language models hinge significantly on how text data is prepared and processed before training. At the heart of this preparation is the process of tokenization, a crucial step where raw text is transformed into a structured format that the models can interpret from. Among the variety of tokenization methods, the sentencepiece tokenizer stands out as a versatile tool that efficiently handles diverse languages without relying on predefined word boundaries.

Tokenization not only facilitates the understanding of language nuances by models but also plays a pivotal role in compressing textual inputs. This compression is essential, as it significantly reduces the dimensionality and complexity of the data, enabling quicker processing and more effective learning. SentencePiece, in particular employs a subword tokenization approach this is adept at managing vocabularies in a way that balances the granularity and the breadth of linguistic elements.

We will focus on the intricacies of tokenizers, with a focus on SentencePiece, exploring their indespensible role in the preprocessing of data for language models. We will discuss the general process of preprocessing, normalizing and post-processing involved in tokenization, shedding light on how these steps contribute to the robuest performance of language models.

Understanding Tokenizers

Tokenizers are the first point of interaction with text data in any nlp system. They transform the raw text into tokens, which are smaller pieces that can be more easily digested by language models. These tokenizers are essentially the building blocks of text analysis and model training. There are several types of tokenizers, each suited to different tasks and languages. Let’s explore the main types and provide examples for each.

Preprocessing

preprocessing is a critical initial step in the text handling pipeline, aimed at preparing and cleaning the text data before it undergoes tokenization and further analysis. The main goal of preprocessing is to standardize the text to reduce variability that is’nt relavent for the subsequent process or analyses. This step ensures that the input to the model is as clean and uniform as possible, improving the model’s ability to learn and make accurate predictions. Here are some key techniques typically employed during the preprocessing phase.

Lowercasing: Converting all characters in the text to lowercase helps in standardizing the data. This means that words like House and house are treated the same preventing the model from treating them as different tokens unnecassarily.
Removing special Characters and punctuation: Text often contains various punctuation marks and special characters, which may not be necassary for many NLP tasks. Removing these elements helps focus the model on the content of words rather than formatting.
Expanding Contractions: In english and many other languages, contractions such as “Can’t”, “don’t”. and “it’s” are common. Expanding these to “cannot” “do not.” and “it is” helps maintain consistency in verb forms and reduces ambiguity in parsing the text.
Removing stopwords: common words such as “and”, “is”, and “but” can be filtered out during preprocessing. These words are usually frequent and carry less meaningful content for many NLP tasks, such as sentiment analysis or topic modelling.
Trimming spaces: Excess whitespaces, including spaces, tabs and new lines can be normalized by trimming them to a single space or removing them entirely, which helps in maintaining consistency in text format

The above text format are not specifically related to sentencepiece model but are some of the common techniques used for preprocessing.

Normalizing

Normalization deals with the way text is represented to ensure consistency. it is particularly important when dealing with different scripts or when the input data comes from various sources that might format text differently.

Unicode Normalization: Converts text to a consistent unicode format, resolving issues like different character encodings for the same characters. Used by BPE, SentencePiece
Stemming and lemmatization: reduces words to their base or root form, either through cutting off inflections (stemming) or by using lexical knowledge of the language (lemmatization).

Byte Pair Encoding (BPE)

BPE is another popular subword tokenization method originally developed for data compression. BPE iteratively merges the most frequent pairs of bytes (or characters) in a dataset to form new tokens, which effectively reduces the vocabulary size and handles the rare words more efficiently.

Vocabulary Independence: While both methods are effective at creating subword vocabularies, SentencePiece does not rely on pre-tokenized text, making it more flexible in handling raw text inputs. BPE typically starts with a base vocabulary of individual characters and builds up by merging frequent pairs.
Handling of Rare Words: Both SentencePiece and BPE help mitigate the issue of rare words through the use of subword units. However, SentencePiece’s algorithm allows for more direct control over the tokenization granularity, which can be advantageous in balancing the vocabulary size against model performance.
Ease of Use: SentencePiece provides an easy-to-use implementation that integrates seamlessly with major machine learning frameworks and supports direct text inputs. BPE often requires initial text processing and vocabulary building, which might add complexity to the pipeline.
Performance in Multilingual Settings: SentencePiece is particularly well-suited for multilingual environments because it treats the text as a sequence of raw Unicode characters, which naturally accommodates multiple languages without bias towards any particular language’s syntax or morphology.

Example

Initial Text

ABABCABCD

Count frequency of pairs

AB: 3 BA: 1 BC: 2 CA: 1 CD: 1

Most frequent pair

AB

Replace ab with a new symbol say `x'

New text: XAXCXCD

Count the frequency of pairs in new text:

XA: 1 AX: 1 XC: 2 CX: 1 CD: 1

Most frequent pair:

XC

Repeat the same process until the desired number of tokens is achieved

SentencePiece Tokenizer

The SentencePiece tokenizer is a robust and flexible tool designed to efficiently manage the tokenization of text without the need for pre-defined word boundaries. This makes it particularly suitable for languages where whitespace is not a reliable delimiter. Unlike traditional tokenization methods that rely on whitespace and pre-defined vocabularies, SentencePiece treats the input text as a raw stream of Unicode characters and learns a vocabulary of subword units directly from this text.

Language Agnosticism: SentencePiece is designed to be independent of the language being processed. It works effectively across various languages, including those that do not use spaces to separate words.
Subword Tokenization: It utilizes subword units, which can effectively capture common prefixes, suffixes, and roots, reducing the out-of-vocabulary issue and preserving meaningful linguistic units.
End-to-End Tokenization: SentencePiece handles both the tokenization and detokenization processes, ensuring that the original text can be perfectly reconstructed from the tokens.

Example

Initial text

this is a simple example

segment the text into characters

t h i s _ i s _ a _ s i m p l e _ e x a m p l e

count the frequency of each character and pair of characters, then merge the most frequent pairs iteratively
assume the vocabulary learned contains

{’th’, ‘is’, ‘_’, ‘a’, ‘si’, ‘mple’, ’ex’, ‘amp’, ’le’}

Tokenize the text using learned vocabulary

Tokenized text: [th, is, _, is, _, a, _, si, mple, _, ex, amp, le]

Why Audio Files Are Really Hard to Compress

Fri, 24 Feb 2023 03:01:20 -0700

Audio data is inherently complex and dense. Unlike text, where redundancy is common (think repeated words or phrases), audio signals are continuous streams of varying frequencies and amplitudes. These signals can include everything from human speech and music to ambient noises and complex soundscapes. The richness and variety in audio data make it challenging to identify and eliminate redundancy without losing essential information.

Human perception of sound

Human ears are sensitive to a wide range of frequencies, from about 20 Hz to 20 kHz. Our perception of sound is also influenced by various factors, such as pitch, volume, and timbre. Any compression algorithm must take these perceptual aspects into account to ensure that the compressed audio still sounds natural to human listeners. This requirement adds an extra layer of complexity to the compression process.

Ineffictive ways we visualize waveforms

Due to The huge amount of data contained in waveforms we try to interpret audio using different techniques like waveforms spectograms these help us get heatmaps of frequency and amplitude distributions over a period of time.

Types of audio compression

Lossless Compression:

Techniques like FLAC (Free Lossless Audio Codec) preserve the exact original audio data, making it possible to reconstruct the original audio perfectly. However, the compression ratios achieved are relatively modest, typically reducing file sizes by about 50% at best.

Lossy Compression:

Formats like MP3 and AAC use psychoacoustic models to remove sounds that are less perceivable to human ears. While these methods achieve higher compression ratios, they do so at the cost of some loss in audio quality. The challenge lies in balancing compression efficiency with audio fidelity.

Applications in Audio Compression

MP3 Encoding:

MP3 compression uses the Modified Discrete Cosine Transform (MDCT), a variation of the Fourier Transform, to convert audio signals into the frequency domain. It then applies psychoacoustic models to remove inaudible frequencies and quantizes the remaining data to achieve compression.

Spectral Analysis:

Fourier Transform is also used in various spectral analysis techniques, which help in identifying and compressing the most important components of the audio signal while discarding redundant or less important parts.