Skip to content

2. Selecting & filtering

Beginner · 9 min read

We'll use the evaluation table from the previous page:

import pandas as pd

df = pd.DataFrame({
    "model":      ["gpt-4o-mini", "llama-3.1-8b", "gpt-4o", "gpt-4o-mini", "llama-3.1-8b", "gpt-4o"],
    "question":   ["What is RAG?", "What is RAG?", "What is RAG?", "Explain agents", "Explain agents", "Explain agents"],
    "latency_ms": [820, 410, 1350, 960, 450, 1500],
    "tokens":     [210, 190, 240, 260, 230, 300],
    "cost_usd":   [0.00013, 0.00002, 0.0024, 0.00016, 0.00002, 0.003],
    "rating":     [5, 4, 5, 4, 3, 5],
})

2.1 Columns

print(df["model"].head(3))                 # one column → Series
print(df[["model", "rating"]].head(3))     # list of columns → DataFrame
Output
0     gpt-4o-mini
1    llama-3.1-8b
2          gpt-4o
Name: model, dtype: str
          model  rating
0   gpt-4o-mini       5
1  llama-3.1-8b       4
2        gpt-4o       5

2.2 Rows with loc (labels) and iloc (positions)

print(df.loc[0])                           # row with index label 0
print(df.loc[1:2, ["model", "latency_ms"]])   # loc slices INCLUDE the end
print(df.iloc[0, 0])                       # first row, first column
print(df.iloc[-2:])                        # last 2 rows (iloc excludes the end)
Output
model          gpt-4o-mini
question      What is RAG?
latency_ms             820
tokens                 210
cost_usd           0.00013
rating                   5
Name: 0, dtype: object
          model  latency_ms
1  llama-3.1-8b         410
2        gpt-4o        1350
gpt-4o-mini
          model        question  latency_ms  tokens  cost_usd  rating
4  llama-3.1-8b  Explain agents         450     230   0.00002       3
5        gpt-4o  Explain agents        1500     300   0.00300       5

loc vs iloc

loc uses labels (index values, column names) and includes the end of a slice. iloc uses positions (0, 1, 2…) and excludes the end — like Python lists.

2.3 Filtering rows with conditions

fast = df[df["latency_ms"] < 900]
print(fast[["model", "latency_ms"]])

good_and_cheap = df[(df["rating"] >= 4) & (df["cost_usd"] < 0.001)]
print(good_and_cheap[["model", "question", "rating"]])
Output
          model  latency_ms
0   gpt-4o-mini         820
1  llama-3.1-8b         410
4  llama-3.1-8b         450
          model        question  rating
0   gpt-4o-mini    What is RAG?       5
1  llama-3.1-8b    What is RAG?       4
3   gpt-4o-mini  Explain agents       4

Use &, |, ~ with brackets

df[df.a > 1 and df.b < 2] raises an error. Write df[(df.a > 1) & (df.b < 2)].

2.4 Handy filters

print(df[df["model"].isin(["gpt-4o", "gpt-4o-mini"])].shape[0], "rows from OpenAI models")
print(df[df["question"].str.contains("agent", case=False)]["model"].tolist())
print(df[df["latency_ms"].between(400, 900)]["latency_ms"].tolist())
Output
4 rows from OpenAI models
['gpt-4o-mini', 'llama-3.1-8b', 'gpt-4o']
[820, 410, 450]

2.5 query — filters as readable text

max_ms = 1000
print(df.query("rating == 5 and latency_ms < @max_ms")[["model", "question"]])   # @ = a Python variable
Output
         model      question
0  gpt-4o-mini  What is RAG?

2.6 Selecting and updating at the same time

df.loc[df["rating"] <= 3, "needs_review"] = True        # new column, set where the condition holds
df["needs_review"] = df["needs_review"].fillna(False).astype(bool)
print(df[["model", "rating", "needs_review"]])
Output
          model  rating  needs_review
0   gpt-4o-mini       5         False
1  llama-3.1-8b       4         False
2        gpt-4o       5         False
3   gpt-4o-mini       4         False
4  llama-3.1-8b       3          True
5        gpt-4o       5         False

Use df.loc[rows, column] = value

Chained assignment like df[df.rating <= 3]["needs_review"] = True changes a copy and leaves df untouched. Always assign through .loc on the original DataFrame.

2.7 Sorting and top N

print(df.sort_values("latency_ms").head(3)[["model", "latency_ms"]])
print(df.sort_values(["rating", "latency_ms"], ascending=[False, True]).head(3)[["model", "rating", "latency_ms"]])
print(df.nlargest(2, "cost_usd")[["model", "cost_usd"]])
Output
          model  latency_ms
1  llama-3.1-8b         410
4  llama-3.1-8b         450
0   gpt-4o-mini         820
         model  rating  latency_ms
0  gpt-4o-mini       5         820
2       gpt-4o       5        1350
5       gpt-4o       5        1500
    model  cost_usd
5  gpt-4o    0.0030
2  gpt-4o    0.0024

Practice

  • Select all rows for llama-3.1-8b with a rating of 4 or more.
  • Show the 3 slowest answers with only the model and latency_ms columns.

Next: Cleaning data →