Confusing these two is common. They serve entirely different purposes:
Game Pak = Book. Contains the story.
Memory Card = Bookmark. Remembers where you are.
Both are essential for a complete N64 experience. Don't skip the memory card if you play games that need it.