The PSP has a fantastic library for kids. All games below are rated E (Everyone) or E10+. Check ESRB descriptors for specifics (cartoon violence, comic mischief).
Top 3 Starter Pack: LittleBigPlanet (creativity), Professor Layton (brains), LocoRoco (pure joy). All E-rated, cheap, defining PSP experiences.