Folio

Source

Atlas Capture Labeling Workshop

38 pages · 13,886 characters

Page 1

Atlas Capture Labeling Workshop This document is the labeling standard for Atlas Capture. Labelers watch first-person (egocentric) video of a person (the ego) and write short command-style captions for each segment so a robot or model can later learn those hand-object skills. Every rule, example, and prohibition in the later pages is binding. If a question is not answered here, the correct reply is that the workshop does not say.

Page 3

What Requires a Label Label Requires - Goal-oriented hand-object actions relevant to the task - Left Hand and Right Hand usage during hand-object interactions Example label: unfold paper with left hand, pick up brush with right hand - Object transfers between hands must be labelled Example label: pass baking tray in right hand to left hand Do NOT label — No Hand-Object interaction, No Label - Walking or navigation - Looking / inspecting / checking - Idle gestures - Camera or face touches - Irrelevant side actions

Page 4

Imperative Voice and Action Separators Imperative Voice Write labels as commands. Correct: pick up spoon with right/left hand Incorrect: the spoon is picked with right/left hand Correct: hold cup with left hand, place screw in box with right hand Incorrect: the screw was placed in box while holding a cup in right hand Action Separators Separate two atomic actions with comma or and. Correct: pick up cup with left hand, place cup on table with left hand Correct: pick up cup with left hand and place cup on table with left hand Incorrect: pick up cup place cup on table with left hand

Page 5

Shape of an Episode Actions observed become separate labels as the goal changes. Example episode 1. Ego picks up a plate while holding a sponge Label: hold sponge with left hand, pick up plate with right hand 2. Ego then opens the faucet Label: hold sponge with left hand, hold plate with right hand, turn on faucet with left hand 3. Ego cleans the plate Label: hold plate with right hand, scrub plate under faucet with sponge in left hand Each segment in the timeline is one self-contained imperative instruction. Do not bleed one action into the neighboring segment.

Page 6

Numericals and Intent Numericals — picking up multiple objects at the same time Incorrect: pick up 3 knives with right/left hand Correct: pick up knives with right/left hand Correct: pick up three knives with right/left hand Holding Multiple Objects Incorrect: hold 2 tools with right/left hand Correct: hold pliers and hammer with right/left hand Intent — Don't label what the ego intends to do Correct: cut tape with scissors with right/left hand Incorrect: pick up scissors to cut tape with right/left hand Correct: tighten screw with screwdriver with right/left hand Incorrect: rotate screwdriver to tighten with right/left hand Correct: twist plastic bag with right/left hand Incorrect: twist plastic bag to close with right/left hand

Page 8

Boundaries (Critical) A segment Starts when the hands begin engaging toward an object. A segment Ends when the hands disengage or the goal changes. Example Timestamp 00:00 to 00:01 — Label: pick up nail polish bottle with left hand Timestamp 00:01 to 00:02 — Label: place nail polish bottle in box with left hand Do not bleed actions into neighboring segments.

Page 9

Timestamp Editing Extend: if the action is cut off in the video, extend the segment until the action is complete. Shorten: if idle or the next action is included, shorten the segment. Use the plus control to extend and the minus control to shorten. The goal is a tight window that covers only the current action.

Page 10

No Action Use No Action only when: - Hands touch nothing for more than 5s - Ego is idle or doing irrelevant behavior for more than 5s No Action Rules - Minor idle time inside a segment is acceptable - Do not split just because the ego pauses Note: An idle period of 5s or less is absorbed into the adjacent segment.

Page 11

Label Length 01 Stay under 20 words 02 All words must be true for the entire segment 03 Max segment duration: 10 seconds; 1 second per label; micro-actions under 1 second round up to 1 second. 04 1-3 atomic actions max per segment Segments should generally contain no more than two conjunction or separator (two commas or the word and). Avoid: "... and ..." chained into a long list of extra actions.

Page 13

Labeling Standards Priority is accuracy, not maximum granularity. A label is a self-contained, imperative instruction that must accurately describe the hand-object interaction(s) in a segment, specified by the ego's hand(s). Dense is preferred but Coarse labels are also allowed provided that there are too many micro actions within a second to label with dense. Accuracy is Key - Be specific - Disambiguate where applicable - Focus on exact actions and objects - Include the tool used and micro-actions - 1-3 atomic actions per label Label Examples pick up shirt from basket and place shirt on table with left hand arrange tarts in left tray with both hands fold shirt with left hand, straighten shirt with right hand pull shirt with both hands, pick up shirt with right hand

Page 14

Labeling Standards — Dense versus Coarse DENSE - Focuses on exact actions and objects - Task contains clear or identifiable actions - Includes micro-actions COARSE - Should only be used when there are too many micro-actions - Focuses on the main goal / objective - As long as within 10s - Either left hand or right hand or both hands

Page 15

Dense Labeling Standard IF DENSE: - Detailed and specific - Focuses on exact actions and objects - Includes the tool used and micro-actions Dense Label Examples pick up shirt with right hand, smoothen shirt with right hand place shirt on table with left hand pick up flat iron with left hand, iron shirt with flat iron in left hand

Page 16

Coarse Labeling Standard IF COARSE: - As long as within 10s and accurate - No micro actions - Focuses on main goal/objective - Includes the tool used - Either left hand or right hand or both hands Coarse Label Examples reposition shirt from basket onto table with left hand hold plastic bag with left hand, fill plastic bag with cat food from container with scooper in right hand smoothen shirt with flat iron in right hand Note: Descriptors required for two or more similar objects when acting on one. When acting on all of them at once, use the collective plural noun — do not enumerate. Correct: pick up tennis balls Incorrect: pick up red and yellow and blue tennis ball

Page 18

Forbidden Words Do not use these words in labels: - adjust — see alternatives (shift, reposition, center, align, slide, level, tilt, unfold, turn, fold, rotate, flatten, straighten, tuck, smoothen, tighten, loosen) - manipulate — see alternatives (grip, hold, push, pull, press, work, twist, flip, squeeze, pinch, apply, assemble) - tool and object — use the actual object name - move, transfer — use pick up and place instead - inspect / check / examine - -ing verbs (present participle) e.g. picking up, placing, working - THE / A / AN e.g. place the cup on table, pick up a shoe - reach — fix timestamps instead - it / them / they (pronouns) e.g. pick up bag with left hand and place it onto table is Incorrect

Page 19

Action Rules | Verb Intuition Object leaves surface → pick up Do not write Pick, Take, or Grasp. Object contacts surface → place Requires a location (general is fine): Incorrect: place cup on upper left of table Correct: place cup on table Relocation with an object → reposition Describing an object being moved from one location to another. Note: Do NOT invent steps.

Page 20

Instead of Adjust use: shift or reposition Examples shift pack on pack with right hand hold cloth with left hand, reposition bag on floor with both hands reposition bag on floor with both hands

Page 21

Instead of Adjust use: center, align, slide, level, tilt Examples center pack on shelf with both hands align cardboard on paper with right hand slide impact drill on table with left hand, pick up metal with right hand level sand mixture in mold with metal scraper in right hand tilt wooden block with left hand, smoothen wooden block with grinder in right hand

Page 22

Instead of Adjust use: unfold, tuck, turn, fold, rotate Examples unfold paper with left hand, pick up brush with right hand tuck garment into hanger clips with both hands turn pages with left hand, hold pages with right hand fold garment on table with both hands rotate screwdriver with right hand

Page 23

Instead of Adjust use: flatten, smoothen, straighten, tighten, loosen Examples flatten dough with rolling pin with both hands smoothen garment on table with both hands straighten garment on table with both hands hold bottle with left hand, tighten lid of bottle with right hand hold bottle with left hand, loosen lid of bottle with right hand

Page 24

Instead of Manipulate use: grip, hold Examples grip storage cart with both hands hold chopsticks with left hand, pick up pot holder with right hand hold rolling pin with left hand, pinch off piece of dough with right hand

Page 25

Instead of Manipulate use: push, pull, press, work Examples push metal containers in bottommost shelf with both hands pull drawer with left hand hold rolling pin with left hand, press dough on table with right hand work dough with both hands

Page 26

Instead of Manipulate use: twist, flip, squeeze, pinch Examples twist dough on mat with both hands flip garment on table with both hands hold plate with left hand, squeeze sponge in right hand pinch off piece of dough with right hand

Page 27

Instead of Manipulate use: apply, assemble Examples apply glue to paper with right hand pick up engine part, assemble engine part with both hands

Page 29

Label Structure Always attach verbs to objects/subjects and always specify the hand used for the action. Acceptable pick up egg with left/right hand, place egg in carton box with left/right hand hold cloth and egg with left/right hand, pick up carton box with left/right hand pick up egg and cloth with left/right hand — use only when ego moves multiple objects at the same time Avoid pick up, place in carton box pick up egg and place in carton box pick up bag, place it onto table

Page 30

Hold Label Always label holding MANIPULATABLE objects; do NOT label holding large stationary objects (tables, walls). Example 1 Ego hold pillow in left hand while cleaning shelf with cloth Correct Segment: hold pillow with left hand, wipe shelf with cloth in right hand Example 2 Ego wipes a cup with a cloth Correct Segment: hold cup with left hand, wipe cup with cloth in right hand Not acceptable: wipe cup with cloth Not acceptable: wipe cup with cloth WITH right hand The held object (tweezers, cup, cloth) is what hold captures. Use "in" for the hand that wields the tool: wipe cup with cloth in right hand. Example 3 Ego wipes a window with a cloth Correct Segment: wipe window with cloth in right hand Not acceptable: hold window with left hand, wipe window with cloth in right hand do NOT label holding large stationary objects (tables, walls).

Page 31

Object Exchanges between Hands Specify the Hand used for the action. Acceptable pick up cup with right hand, hand over cup in right hand to left hand pick up cup with right hand, put cup in right hand to left hand pick up cup with right hand, pass cup in right hand to left hand Also applicable: switch, set Wrong pick up cup with right hand, transfer cup in right hand to left hand pick up cup with right hand, handover cup in right hand to left hand pick up cup with right hand, give cup in right hand to left hand

Page 32

Split and Merge Segment Split (when) - Hands disengage - A new goal / objective begins - Exceeds 10 seconds Merge (when allowed) - Group repeated same-object interactions (5+ times within 10s) - Group rapid interactions across multiple objects (example: sift through laundry basket with both hands)

Page 33

Repeated Actions Rule 1 If the ego stops and then starts again for less than 5x, treat them as separate segments. Rule 2 If the ego never disengages, it is one segment unless more than 10s.

Page 34

Descriptors - Descriptors are required for two or MORE similar objects acting on one - When acting on ALL of them at once, use the collective plural noun — do not enumerate Position Left / right / upper / bottom only to disambiguate similar objects (from ego's perspective but describing object based on attributes is preferred). Correct: fold sides of shirt with both hands Correct: hold shoe with left hand, scrub bottom part of shoe with brush in right hand Adjectives Use adjectives only to disambiguate two or more similar items. Correct: blue cloth vs white cloth Otherwise keep it simple: pick up cloth with left hand Body Parts Avoid body parts unless unavoidable (with the exception of hand). Preferred: place fabric on surface with right hand Acceptable if needed: place fabric on lap with right hand

Page 35

Descriptors — Object-centric versus Egocentric Object-Centric Perspective (movable objects) Color based Identifiers — pick up blue cloth with left hand Pattern or Appearance — pick up striped cloth with left hand Relationship to nearby objects — pick up marker beside notebook with left hand Container or Location based — pick up spoon in sink with right hand Size-Based Identifiers — pick up large bowl with both hands State-Based Identifiers — pick up wrapped package with both hands Function-Based Identifiers — pick up serving spoon with right hand Egocentric Perspective (large stationary objects) Example 1 Ego wipe two cabinets — wipe cabinet with cloth in right hand Example 2 Ego wipes sofa — wipe armrest of sofa with cloth in both hands Example 3 Ego wipes two shelves — place box on shelf with both hands

Page 37

Ideal Segment vs Audit Fail Ideal Segment checklist - Full action coverage - Accurate verbs - No hallucinated actions Audit Fail conditions - Missed action - Missed hands used for action - Hallucination - Segment bleeding (timestamps cut off or include next action) - Forbidden words (the, a, an, -ing) - No-action label combined with action (example: no action, place cup on table)

Clips

Upload screenshots or a short video. Folio applies only this source — including hand, verb, and forbidden-word rules.