Collapse transcript

00:00 Hello and welcome to Python Bytes, where we deliver Python news and headlines directly to

00:05 your earbuds. This is episode 56, recorded December 12th, 2017. I'm Michael Kennedy.

00:11 And I'm Brian Okken.

00:12 And we got a bunch of awesome stuff like always lined up for you. Before we get to it, Brian,

00:16 let's just say thanks to Rollbar.

00:17 Thanks, Rollbar.

00:17 Yeah, thank you, Rollbar, for supporting the show, tell you more about what they got going

00:21 on later. I want to hear the tick-tock of the pendulum, if you will.

00:26 I was reading an article called Dealing with Date Times Like a Pro in Python.

00:30 Which can be way harder than it should be, right?

00:32 Yes. And I think that there's things like Arrow and Maya that try to get so that, you know,

00:38 I really have to deal with date times directly. But this is actually a nice article on dealing

00:43 with date times if you want to use the direct library. However, at the end of it, they mentioned

00:48 Pendulum, which I hadn't run across before. So checking it out, there's a link here for the

00:53 Pendulum Python Date Times Made Easy. And the website is beautiful. But also the interface

01:01 to it, it looks really nice. I mean, you can do things like just import Pendulum and then

01:08 Pendulum Now in Europe, Paris, or in America, or something. And then you can do things like

01:15 add dates together and do, you know, subtract dates. And the normal things you have to do with

01:21 date times. But the just the model looks nice. I like it.

01:24 Yeah, it looks really, really nice. And the website is quite stunning. And you know, to me,

01:30 like on one hand, like, okay, whatever, it's documentation. On the other, it really shows the

01:34 people working on this actually care about this project. And it's not just some random thing

01:38 thrown on GitHub. So yeah, it's really cool. So you say like Pendulum.now and you give it a time

01:42 zone, say like Europe slash Paris. And then you can subtract weeks from it. Or you can,

01:47 one thing I like to do is put times relative to humans like a week ago, five seconds ago in 10

01:53 minutes. And so it just has a diff for humans, you know, and that's really nice.

01:58 I'm going to include in the show notes links to Arrow and Maya. And I think it's, it's a matter of

02:03 developer taste really, and what looks more natural to you. So check them all out and see what's nicer

02:09 for you.

02:09 Yeah, it's really nice. The other one to throw in here while we're thinking about all this is parsing

02:13 date times, which I think there's 700 different formats for date times and text, which is insane.

02:19 But I really like to use Python date util. So pip install Python dash date util, but just import

02:25 date util. And it compars like lots and lots of stuff. So you can just go here, load that into time,

02:31 you know, and that's, that's one more thing. But this is really, really slick. It has more

02:35 sort of time delta things like how many weeks, how many days, how many hours, as opposed to just like

02:40 the seconds, total seconds, the built in one really nice. I like it. Good find. So the next thing that

02:45 I want to talk about, we're going to talk about some web stuff, aren't we? So the next one I want

02:48 to do is, it's sort of a web thing, but it doesn't really have to be it's any kind of service or long

02:53 running thing. So one of the things that can be a challenge with web applications is long, long

03:00 running tasks or tasks that are slow, but you don't really need to like wait on their outcome.

03:04 So I'll give you an example. On my training website, training.talkpython.fm on that site,

03:10 there's a capability to email everyone in a class. And it turns out, if you in the same request,

03:19 actually email thousands of people, one at a time, like through Amazon SES or something that

03:25 request times out, and then you've emailed some people and you don't know how far you got,

03:31 you've got to go figure out how to rewrite that some other way, right? That's a problem.

03:35 Yeah. So there's certain things like email, like logging, like other sort of kick this job off and

03:41 forget about it, that would be much better served in some sort of asynchronous way. One of the best

03:46 things you can do to add scalability to your app is to push those kinds of things onto some background,

03:50 sort of get to it when you can style. And probably the most formal way is to use some kind of queuing.

03:57 So this, the thing I want to talk about now is this article called Flask, asynchronous background

04:01 tasks with Celery and Redis. Although it has nothing really to do with Flask, although it just

04:06 uses Flask as an example. It could be Django, it could be Pyramid, like really it's the Celery bit

04:10 that's interesting. So you take Celery as the work, the worker, right? It's going to run back there.

04:15 It's going to look at the Redis message bus. And what you do is you just kick jobs off to it,

04:21 and it'll just start pulling them off and working on them. So what I could have done is said,

04:24 I want to email, you know, 3000 people. So go to the database, get the 3000 people and just put 3000

04:30 email this person jobs onto a Celery queue and been done straight away. And it would just, you know,

04:36 do it over the next minute or two. Actually, that's pretty cool. Yeah, I've got my the wheels are

04:40 turning in my head thinking things that I want to use Celery for. I know, I feel the same way. It's

04:44 really nice write up of how you like walk through these things. So it talks about using the Celery client to

04:48 connect Flask to Celery, the Celery worker process, the message broker that works with the

04:54 Redis back end. And the other thing that's interesting about this is so often these Python

04:59 examples, these especially Python web examples are done on macOS or Linux, and this is all on Windows.

05:05 And I don't think Redis is even supported on Windows or Celery, one of them. I think it's Redis. It's not,

05:11 there's like a, an unofficial build or something. So I think it's kind of extra interesting to show

05:16 this other angle and show people how to do it where it might not be so obvious. Also something

05:21 like taking on Celery, it's kind of nice to look at an example project to see how it's been used,

05:26 because just jumping right into the entry of the documentation can be a little scary sometimes.

05:31 Here's a nice little simple example and take it and do what you will with it. Yeah. Well,

05:35 we have another web example that I liked in it. This is from Bob Belderboss of PyBytes. He wrote a

05:43 guest article on real Python called building a simple web app with bottle SQLAlchemy and the

05:49 Twitter API. And the reason why this caught my attention is because I was curious about having some

05:55 way to interact with Twitter and Python. So I think I'm going to walk through this a little bit. And

06:00 from start to finish, it's a project that pulls, it isn't pushing things into Twitter,

06:06 it's reading Twitter and then displaying them on a, a little bottle website that they pushed up to

06:12 Heroku. And that's kind of cool that they document everything from start to finish for this.

06:17 Yeah. And it has the deployment bit as well, like pushing it to Heroku and making it work there,

06:22 which is also kind of cool because your web apps are fun, but they belong on the web, not your laptop.

06:27 It's on my queue of things to, on my salary queue of things to learn because I'm, I'm busy learning

06:33 Flask at the time, but maybe I'll check this out next. Yeah. It sounds really interesting. Definitely.

06:38 Nice job, Bob. Now, before we get on to talking about something that's actually not on the web,

06:45 but to do with Python, let me just tell you a quick story of yesterday. Yesterday I decided,

06:52 let me see how far I can push the performance of uWSGI and NGINX on the talk Python sites.

06:59 Luckily for you, Brian, I didn't do this on pythonbytes.fm, but like, all right, I think I can push

07:06 the parallelism out better. I could crank up the number of threads. I can crack open the process

07:11 affinity, processor affinity. So like get the maximum parallelization out of uWSGI, which is one of

07:18 the better sort of application tier worker process web servers for Python. Okay. So that seemed like a

07:26 good idea, right? Yeah. Turns out something about the threading was bad and I don't know what it is.

07:32 I don't really care to look into it, but I started getting roll bar notifications saying that the template

07:40 files could not be found for the website, which has been basically largely unchanged for months.

07:48 And then it couldn't convert this template to a string and all sorts of random errors. And I'm like,

07:54 what is going on? And there's some kind of corruption timing problem in this whole series.

07:59 And I would have, it happened like one out of a hundred thousand times. I mean, really,

08:04 really rare, but it started coming in like a couple of times yesterday. So I'm like, all right, well,

08:09 I guess we're going to turn that back down and turn it back down. Errors went away. And without roll bar,

08:13 I never would have known it. So if you guys want to make sure that your apps are working,

08:17 go check out roll bar, just pythonbytes.fm/rollbar and you sign up and you too may start getting

08:24 weird messages that you need to go look into. It'd be awesome. Yeah. Nice.

08:27 Yeah. So that was yesterday, but today I want to move on and talk about something that actually

08:34 covered way back on Talk Python, let's say about a 40 episodes ago. So not quite a year, 40 weeks ago.

08:41 And that was the Visual Studio Code extension written by a guy named Don Jayamana.

08:48 John Amani. Sorry about that. I'm sure I'm messing up your name. And he wrote a really cool Python

08:55 extension to basically add Python capabilities to VS Code. And it turned out to be the first or second

09:00 most popular thing in Visual Studio Code. It happened like millions of downloads. That's pretty cool,

09:05 right? That's very cool. Yeah. Yeah. And so one of the things I looked at, you know, when we spoke,

09:10 it's pretty interesting. He took, I'd say maybe 10 different packages, Jedi and other, you know,

09:16 Flake 8 and those types of things. And he basically wove those together into an extension for VS Code.

09:22 He didn't really build a huge amount, but he definitely put it together in a really, really usable way.

09:27 So that was, that was pretty cool. So today, this week, the news is that he is now officially hired by

09:34 Microsoft because of his open source project. That's nice. Really cool. Yeah. So here's a guy who started

09:39 an open source project and got this pretty influential job just because he had a really successful

09:45 plugin for this open source thing. So pretty, pretty cool. You know, it doesn't really mean many changes.

09:50 I just, there's a few more people working on the Python extension for VS Code, which if you're into VS Code,

09:56 it's a lot like Adam or a sublime. It's quite, quite nice. If that's your style of editor, it does a lot of

10:03 cool stuff. And so basically there'll be more people working on it, fewer bugs, faster turnaround.

10:07 The other big news is they're also hiring for a Visual Studio Code Python developer. So if you're

10:12 looking for a job and that sounds pretty awesome, you know, there's some really cool guys over there.

10:16 There's Brett Cannon, there's Steve Dower, some of the big hitters on the Python core dev team,

10:21 in addition to Don. So that might be worth looking into if that interests you.

10:25 Nice. So I feel like you're kind of doing this web crash course and getting into a lot of different

10:30 things on the web, right?

10:32 I'm trying to embrace the web a little bit more.

10:34 Nice. So what'd you find?

10:35 Yeah. So one of the things with, we talk about a lot of mechanics of how to get websites up

10:41 with Flask and Bottle and Django and all that. But sometimes the design of your website that,

10:49 as it turns out at the end, is like leaves a little bit lacking.

10:53 It looks like a developer designed it.

10:55 Yeah, it definitely looks like a developer designed it, which is usually better than when it looks like

11:01 like a database engineer designed it. But still, there's an article called a comprehensive,

11:07 it's on Smashing Magazine called a comprehensive guide to web design. It's a long article, but it's

11:13 all one page. And it's a, it's a nice, I think a good overview of the basically the basics of

11:20 make sure you have this stuff down and to make sure your website doesn't suck too bad.

11:26 That's really cool. I feel like design is one of those things that feels like a barrier to developers.

11:31 You're like, I'm not a designer. I can't do stuff that's pretty. I can't build these types of things.

11:35 But if you put in just a little effort, you can turn that barrier into a superpower because most

11:39 people don't have it. And it's actually not that hard if you just sit down and focus and try to study it

11:45 for a month or something. Especially a lot of times, like there's a lot of projects that get done, like

11:49 internal work projects or, or different tools for a small group of people that you don't have the money

11:56 to hire a front end designer. So something like this to go through and make sure that you do your breadcrumbs

12:02 appropriately and you put your search box in a findable place and some of these page structure

12:09 are reviews that are good to just make, make it more usable. And it, this doesn't add a lot of work to your

12:14 workflow. It actually, I think it takes away some of the stress of designing a web page if you're not a designer.

12:21 Yeah, it's cool. It gives you a framework to think through and stuff you got to do anyway. That's cool.

12:24 All right. Final item. What would you name the baby of requests and Selenium coming together to create some new library?

12:33 Well, I know the answer, but.

12:35 Requestium.

12:36 Requestium.

12:37 Maybe, maybe one of the candidates you throw out there, right?

12:40 Yeah.

12:40 So the thing I want to talk about is this thing called Requestium. And we have requests,

12:44 which everybody knows, super, super popular library for just downloading HTML.

12:49 And we have Selenium, which fewer people know, but is basically a headless Chrome browser.

12:55 There's other options as well, but let's say Chrome for now. And you control it from Python.

12:59 And so use them in different situations. Requests is super lightweight, takes no, very,

13:04 very few dependencies. I mean, it doesn't depend on say Chrome. It just depends on a few packages.

13:08 And you can download the HTML and do what you will with it. But if you happen to run into,

13:13 let's say an AngularJS or VueJS or any other JS site that has like a front end JavaScript framework

13:20 that basically requires the JavaScript to execute, to get the content, you end up with just a bunch

13:26 of curly braces that have no meaning when you hit it with requests, right?

13:29 Yeah. I didn't know that.

13:30 Yeah. You just get the HTML that's basically, here's where JavaScript will fill out the holes.

13:34 Good luck with that. You know what I mean? And so that's kind of useless. So you might use Selenium,

13:39 but you know, Selenium is way more heavyweight. So that Requestium basically merges the power of

13:44 requests, Selenium, and this thing called Parcel, which is like beautiful soup or Scrapey. It

13:49 understands HTML and lets you ask questions of it. So the idea is like, it's written for mostly when

13:55 you're doing requests, but that one time or two that you're like, oh, you need this login sequence

14:00 has to happen in a richer JavaScript way. And it lets you do crazy things like automatically switch

14:06 between request sessions and Selenium sessions. And it lets you basically, I think, share cookies

14:13 across them and all sorts of interesting stuff like that. So if you need like a little bit heavier weight,

14:18 then request, but not full on Selenium, here's a nice combo sort of marriage of the two.

14:23 Oh, that's nice. Cool.

14:24 Yeah. So very good for either testing or web scraping or some other, other types of things.

14:30 But if you need to test something that's JavaScript heavy, this is a pretty good,

14:33 pretty good option here.

14:34 Nice.

14:34 All right. Well, that's our news, Brian, at least the ones we've selected for everyone else. But how

14:38 about yourself? What do you got going on?

14:39 Well, I finally got a, a testing code episode out this not too long ago. I don't know if I brought it

14:45 up already.

14:46 I don't think so. Not here.

14:47 It was a nice interview with Catherine Jarmal or Kjam about testing and data science. And it's a

14:55 testing code episode 33, and it's been very well received so far. So a lot of people get some value

15:00 out of it.

15:01 Yeah, that's great. Catherine's done a bunch of cool work. So definitely go over there and check it out.

15:04 And the last thing I wanted to mention was I had asked people to leave Amazon reviews for books

15:10 that they like, including mine, hopefully. And so far, nine people have left reviews on Python testing

15:17 with pytest. So thanks a lot.

15:19 Yeah, that's great.

15:19 How about you? Any news?

15:20 Not a whole lot of news. I'm working hard, but I'm getting things ready to talk about. Nothing

15:26 to talk about yet, but someday soon. I guess the big thing I have coming up, I just recorded

15:31 some nice talk Python episodes. One actually on optimizing web servers. And then later this week,

15:38 I'm recording one, which is a panel session on machine learning and artificial intelligence at

15:44 the Large Hadron Collider and in particle physics.

15:46 Oh, that sounds interesting. And the last few episodes that you've put out, I mean, you always

15:52 have good episodes, but I've really enjoyed the last few quite a bit.

15:54 Thanks a bunch. Yeah, same for you. All right, man. Well, thank you for sharing this stuff. And

16:00 everyone, thank you for listening.

16:01 Thank you.

16:02 Thank you for listening to Python Bytes. Follow the show on Twitter via @pythonbytes.

16:07 That's Python Bytes as in B-Y-T-E-S. And get the full show notes at pythonbytes.fm.

16:14 If you have a news item you want featured, just visit pythonbytes.fm and send it our way. We're

16:18 always on the lookout for sharing something cool. On behalf of myself and Brian Okken,

16:23 this is Michael Kennedy. Thank you for listening and sharing this podcast with your friends and colleagues.