As Microsoft’s new Xbox One game console shows up in homes across the country this holiday season, gamers are high-fiving. But their siblings, spouses and roommates who aren’t into videogames are asking, “What’s in it for me?”

For the past week, I’ve been testing the Xbox One from their perspective, and there’s a lot for non-gamers. The name comes from Microsoft’s intention that it become the one central box for all of your digital living-room activities without the hassle of changing inputs. You can use it for watching TV, conducting Skype video calls, browsing the Web, watching streamed videos, opening files and photos, fitness and, of course, games. Xbox One comes with Kinect, the seemingly magic sensor that responds to your gestures and voice commands, letting you navigate without a controller.

But competition for space in the living room is fierce. In addition to Sony’s new $400 PlayStation 4, Xbox must now compete against tablets people hold on their laps, as well as features in popular devices like the $99 Apple TV and $50 Roku LT, which take up a tiny square of counter space.

The $500 Xbox One is massive in comparison, measuring about 13 inches by 11 inches and weighing nearly eight pounds. Then there’s the nearly 10-inch-wide Kinect sensor. What’s worse: Xbox One comes with a gamer-centric controller that design-conscious homeowners will want to hide in a drawer.

After using the Xbox One regularly in my home, I can say that several of its features will appeal to non-gamers. Its Kinect camera made my Skype video calls look and sound remarkably good. Apps downloaded from the Xbox Store in seconds, and I quickly shifted from apps to videos to TV. A Snap feature lets you slide one app, video or game to the side of the screen while watching something else in the fuller part of the screen. And when it works, voice control with the Kinect can be very cool.

Yet a lot of things went wrong. On the fourth day of use, my Kinect stopped working altogether. After troubleshooting with Microsoft, I had to get a new Xbox One. Microsoft’s chief marketing and strategy officer, Yusuf Mehdi, said he hadn’t yet heard of this problem happening.

When my Kinect was working, its voice commands left a lot to be desired — especially when I tried giving commands over noise in the room. When I set up the second Xbox One, I even tried an extra step in calibrating Kinect’s microphone, but this didn’t make a significant difference. Despite on-screen suggestions for specific voice commands, users must remember far too many commands.

While watching TV with the Xbox One, you can change the channel using some simple commands like, “Watch the Food Network,” though you might be like me and forget what channel hosts each show. Saying “What’s on?” prompts a list of channels so you can know what to say. To skim up and down in the One Guide, I said, “Page up” to go slowly or “Jump up” to move farther up the guide. Volume can be turned up or down, but each vocal prompt only adjusts volume by a few notches, which gets frustrating when you have to keep saying, “Volume down” over and over.

Another problem was that though my TV channels were pulled into the Xbox One (by plugging in a cable box) and handsomely displayed in the One Guide, I still had to pick up a TiVo remote to access and play back my recorded TV content. If you’re like me, and watch recorded shows more often than live TV, shifting remotes can be a significant source of frustration.

As is true with all voice-command products, talking to get something done can feel rude if you have to interrupt a conversation. This happened several times in my living room.

But for all of its flaws, as soon as my Kinect stopped working, I missed using voice commands. I realized how much I was using (or trying to use) them before anything else. Microsoft says voice recognition will improve over time, so there’s hope for more accuracy in the future.

My initial setup of the Xbox One console and Kinect was straightforward and user-friendly. It asked me to sign in with my Microsoft ID and password and let me select a color to represent my account. The Kinect setup involved a quick process of confirming that its camera worked and calibrating the microphone.

After this, the Kinect recognized me any time I walked into my living room, and said hi with a small pop-up icon at the bottom of the screen when the TV was on. The same was true for my husband, who had his own account. This is helpful if each family member has different settings, like favorite channels or apps pinned to your Xbox Home screen.

I downloaded and used the NFL app, Skype, SkyDrive, Xbox Music, Amazon Instant Video and others. I opened the NFL app during the Giants versus Cowboys game and snapped it to the right so I could watch the game while keeping that app to one side.

I browsed the Web using Internet Explorer, reading a few articles on WSJ.com and watching their accompanying videos. A precise on-screen cursor in the shape of a circle with a point made Web browsing easier than I expected.

A Skype video call to my mom prompted a “Wow!” from her as she admired the quality of the picture, which she described as amazing. As for audio, she said she could hear me just as clearly when the loud, humming central heat clicked on in my house. As I moved around the living room and talked to my mom from six different places, the Kinect camera panned to follow me, and even zoomed in on my face for the best possible picture. I had a similarly good experience during a call to someone else who was using the Xbox One’s new Kinect.

Though the Xbox One is still geared toward gamers, it will appeal to a broader audience with its variety of apps and ways of watching TV. Just be ready for a potentially frustrating experience when you try talking to the Xbox One.

Write to Katie at katie.boehret@wsj.com