Python <- StackOverflow top 100

Question

I am a bit puzzled by the following code:

d = {'x': 1, 'y': 2, 'z': 3} 
for key in d:
    print key, 'corresponds to', d[key]

What I don’t understand is the key portion. How does Python recognize that it needs only to read the key from the dictionary? Is key a special word in Python? Or is it simply a variable?

Answer accepted (score 5011)

key is just a variable name.

for key in d:

will simply loop over the keys in the dictionary, rather than the keys and values. To loop over both key and value you can use the following:

For Python 2.x:

for key, value in d.iteritems():

For Python 3.x:

for key, value in d.items():

To test for yourself, change the word key to poop.

For Python 3.x, iteritems() has been replaced with simply items(), which returns a set-like view backed by the dict, like iteritems() but even better. This is also available in 2.7 as viewitems().

The operation items() will work for both 2 and 3, but in 2 it will return a list of the dictionary’s (key, value) pairs, which will not reflect changes to the dict that happen after the items() call. If you want the 2.x behavior in 3.x, you can call list(d.items()).

Answer 2 (score 401)

It’s not that key is a special word, but that dictionaries implement the iterator protocol. You could do this in your class, e.g. see this question for how to build class iterators.

In the case of dictionaries, it’s implemented at the C level. The details are available in PEP 234. In particular, the section titled “Dictionary Iterators”:

• Dictionaries implement a tp_iter slot that returns an efficient iterator that iterates over the keys of the dictionary. […] This means that we can write

  for k in dict: ...

  which is equivalent to, but much faster than

  for k in dict.keys(): ...

  as long as the restriction on modifications to the dictionary (either by the loop or by another thread) are not violated.

• Add methods to dictionaries that return different kinds of iterators explicitly:

  for key in dict.iterkeys(): ...
  
  for value in dict.itervalues(): ...
  
  for key, value in dict.iteritems(): ...

  This means that for x in dict is shorthand for for x in dict.iterkeys().

In Python 3, dict.iterkeys(), dict.itervalues() and dict.iteritems() are no longer supported. Use dict.keys(), dict.values() and dict.items() instead.

Answer 3 (score 189)

Iterating over a dict iterates through its keys in no particular order, as you can see here:

Edit: (This is no longer the case in Python3.6, but note that it’s not guaranteed behaviour yet)

>>> d = {'x': 1, 'y': 2, 'z': 3} 
>>> list(d)
['y', 'x', 'z']
>>> d.keys()
['y', 'x', 'z']

For your example, it is a better idea to use dict.items():

>>> d.items()
[('y', 2), ('x', 1), ('z', 3)]

This gives you a list of tuples. When you loop over them like this, each tuple is unpacked into k and v automatically:

for k,v in d.items():
    print(k, 'corresponds to', v)

Using k and v as variable names when looping over a dict is quite common if the body of the loop is only a few lines. For more complicated loops it may be a good idea to use more descriptive names:

for letter, number in d.items():
    print(letter, 'corresponds to', number)

It’s a good idea to get into the habit of using format strings:

for letter, number in d.items():
    print('{0} corresponds to {1}'.format(letter, number))

Question

I’m looking for a string.contains or string.indexof method in Python.

I want to do:

if not somestring.contains("blah"):
   continue

Answer accepted (score 5757)

You can use the in operator:

if "blah" not in somestring: 
    continue

Answer 2 (score 601)

If it’s just a substring search you can use string.find("substring").

You do have to be a little careful with find, index, and in though, as they are substring searches. In other words, this:

s = "This be a string"
if s.find("is") == -1:
    print "No 'is' here!"
else:
    print "Found 'is' in the string."

It would print Found 'is' in the string. Similarly, if "is" in s: would evaluate to True. This may or may not be what you want.

Answer 3 (score 161)

if needle in haystack: is the normal use, as @Michael says – it relies on the in operator, more readable and faster than a method call.

If you truly need a method instead of an operator (e.g. to do some weird key= for a very peculiar sort…?), that would be 'haystack'.__contains__. But since your example is for use in an if, I guess you don’t really mean what you say;-). It’s not good form (nor readable, nor efficient) to use special methods directly – they’re meant to be used, instead, through the operators and builtins that delegate to them.

Question

How do I see if a file exists or not, without using the try statement?

Answer 2 (score 4830)

If the reason you’re checking is so you can do something like if file_exists: open_it(), it’s safer to use a try around the attempt to open it. Checking and then opening risks the file being deleted or moved or something between when you check and when you try to open it.

If you’re not planning to open the file immediately, you can use os.path.isfile

Return True if path is an existing regular file. This follows symbolic links, so both islink() and isfile() can be true for the same path.

import os.path
os.path.isfile(fname) 

if you need to be sure it’s a file.

Starting with Python 3.4, the pathlib module offers an object-oriented approach (backported to pathlib2 in Python 2.7):

from pathlib import Path

my_file = Path("/path/to/file")
if my_file.is_file():
    # file exists

To check a directory, do:

if my_file.is_dir():
    # directory exists

To check whether a Path object exists independently of whether is it a file or directory, use exists():

if my_file.exists():
    # path exists

You can also use resolve(strict=True) in a try block:

try:
    my_abs_path = my_file.resolve(strict=True)
except FileNotFoundError:
    # doesn't exist
else:
    # exists

Answer 3 (score 1993)

You have the os.path.exists function:

import os.path
os.path.exists(file_path)

This returns True for both files and directories but you can instead use

os.path.isfile(file_path)

to test if it’s a file specifically. It follows symlinks.

Question

In Python, how can I parse a numeric string like "545.2222" to its corresponding float value, 545.2222? Or parse the string "31" to an integer, 31?

I just want to know how to parse a float str to a float, and (separately) an int str to an int.

Answer accepted (score 2443)

>>> a = "545.2222"
>>> float(a)
545.22220000000004
>>> int(float(a))
545

Answer 2 (score 487)

def num(s):
    try:
        return int(s)
    except ValueError:
        return float(s)

Answer 3 (score 477)

Python method to check if a string is a float:

def is_float(value):
  try:
    float(value)
    return True
  except:
    return False

A longer and more accurate name for this function could be: is_convertible_to_float(value)

What is, and is not a float in Python may surprise you:

val                   is_float(val) Note
--------------------  ----------   --------------------------------
""                    False        Blank string
"127"                 True         Passed string
True                  True         Pure sweet Truth
"True"                False        Vile contemptible lie
False                 True         So false it becomes true
"123.456"             True         Decimal
"      -127    "      True         Spaces trimmed
"\t\n12\r\n"          True         whitespace ignored
"NaN"                 True         Not a number
"NaNanananaBATMAN"    False        I am Batman
"-iNF"                True         Negative infinity
"123.E4"              True         Exponential notation
".1"                  True         mantissa only
"1,234"               False        Commas gtfo
u'\x30'               True         Unicode is fine.
"NULL"                False        Null is not special
0x3fade               True         Hexadecimal
"6e7777777777777"     True         Shrunk to infinity
"1.797693e+308"       True         This is max value
"infinity"            True         Same as inf
"infinityandBEYOND"   False        Extra characters wreck it
"12.34.56"            False        Only one dot allowed
u'四'                 False        Japanese '4' is not a float.
"#56"                 False        Pound sign
"56%"                 False        Percent of what?
"0E0"                 True         Exponential, move dot 0 places
0**0                  True         0___0  Exponentiation
"-5e-5"               True         Raise to a negative number
"+1e1"                True         Plus is OK with exponent
"+1e1^5"              False        Fancy exponent not interpreted
"+1e1.3"              False        No decimals in exponent
"-+1"                 False        Make up your mind
"(1)"                 False        Parenthesis is bad

You think you know what numbers are? You are not so good as you think! Not big surprise.

Don’t use this code on life-critical software!

Catching broad exceptions this way, killing canaries and gobbling the exception creates a tiny chance that a valid float as string will return false. The float(...) line of code can failed for any of a thousand reasons that have nothing to do with the contents of the string. But if you’re writing life-critical software in a duck-typing prototype language like Python, then you’ve got much larger problems.

Question

How can I list all files of a directory in Python and add them to a list?

Answer 2 (score 3664)

os.listdir() will get you everything that’s in a directory - files and directories.

If you want just files, you could either filter this down using os.path:

from os import listdir
from os.path import isfile, join
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

or you could use os.walk() which will yield two lists for each directory it visits - splitting into files and dirs for you. If you only want the top directory you can just break the first time it yields

from os import walk

f = []
for (dirpath, dirnames, filenames) in walk(mypath):
    f.extend(filenames)
    break

Answer 3 (score 1475)

I prefer using the glob module, as it does pattern matching and expansion.

import glob
print(glob.glob("/home/adam/*.txt"))

It will return a list with the queried files:

['/home/adam/file1.txt', '/home/adam/file2.txt', .... ]

Question

For a list ["foo", "bar", "baz"] and an item in the list "bar", how do I get its index (1) in Python?

Answer accepted (score 4050)

>>> ["foo", "bar", "baz"].index("bar")
1

Reference: Data Structures > More on Lists

Caveats follow

Note that while this is perhaps the cleanest way to answer the question as asked, index is a rather weak component of the list API, and I can’t remember the last time I used it in anger. It’s been pointed out to me in the comments that because this answer is heavily referenced, it should be made more complete. Some caveats about list.index follow. It is probably worth initially taking a look at the docstring for it:

>>> print(list.index.__doc__)
L.index(value, [start, [stop]]) -> integer -- return first index of value.
Raises ValueError if the value is not present.

Linear time-complexity in list length

An index call checks every element of the list in order, until it finds a match. If your list is long, and you don’t know roughly where in the list it occurs, this search could become a bottleneck. In that case, you should consider a different data structure. Note that if you know roughly where to find the match, you can give index a hint. For instance, in this snippet, l.index(999_999, 999_990, 1_000_000) is roughly five orders of magnitude faster than straight l.index(999_999), because the former only has to search 10 entries, while the latter searches a million:

>>> import timeit
>>> timeit.timeit('l.index(999_999)', setup='l = list(range(0, 1_000_000))', number=1000)
9.356267921015387
>>> timeit.timeit('l.index(999_999, 999_990, 1_000_000)', setup='l = list(range(0, 1_000_000))', number=1000)
0.0004404920036904514

Only returns the index of the first match to its argument

A call to index searches through the list in order until it finds a match, and stops there. If you expect to need indices of more matches, you should use a list comprehension, or generator expression.

>>> [1, 1].index(1)
0
>>> [i for i, e in enumerate([1, 2, 1]) if e == 1]
[0, 2]
>>> g = (i for i, e in enumerate([1, 2, 1]) if e == 1)
>>> next(g)
0
>>> next(g)
2

Most places where I once would have used index, I now use a list comprehension or generator expression because they’re more generalizable. So if you’re considering reaching for index, take a look at these excellent python features.

Throws if element not present in list

A call to index results in a ValueError if the item’s not present.

>>> [1, 1].index(2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: 2 is not in list

If the item might not be present in the list, you should either

1. Check for it first with item in my_list (clean, readable approach), or
2. Wrap the index call in a try/except block which catches ValueError (probably faster, at least when the list to search is long, and the item is usually present.)

Answer 2 (score 861)

One thing that is really helpful in learning Python is to use the interactive help function:

>>> help(["foo", "bar", "baz"])
Help on list object:

class list(object)
 ...

 |
 |  index(...)
 |      L.index(value, [start, [stop]]) -> integer -- return first index of value
 |

which will often lead you to the method you are looking for.

Answer 3 (score 511)

The majority of answers explain how to find a single index, but their methods do not return multiple indexes if the item is in the list multiple times. Use enumerate():

for i, j in enumerate(['foo', 'bar', 'baz']):
    if j == 'bar':
        print(i)

The index() function only returns the first occurrence, while enumerate() returns all occurrences.

As a list comprehension:

[i for i, j in enumerate(['foo', 'bar', 'baz']) if j == 'bar']

Here’s also another small solution with itertools.count() (which is pretty much the same approach as enumerate):

from itertools import izip as zip, count # izip for maximum efficiency
[i for i, j in zip(count(), ['foo', 'bar', 'baz']) if j == 'bar']

This is more efficient for larger lists than using enumerate():

$ python -m timeit -s "from itertools import izip as zip, count" "[i for i, j in zip(count(), ['foo', 'bar', 'baz']*500) if j == 'bar']"
10000 loops, best of 3: 174 usec per loop
$ python -m timeit "[i for i, j in enumerate(['foo', 'bar', 'baz']*500) if j == 'bar']"
10000 loops, best of 3: 196 usec per loop

Question

How do I read every line of a file in Python and store each line as an element in a list?

I want to read the file line by line and append each line to the end of the list.

Answer 2 (score 2035)

with open(filename) as f:
    content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content] 

Answer 3 (score 915)

See Input and Ouput:

with open('filename') as f:
    lines = f.readlines()

or with stripping the newline character:

lines = [line.rstrip('\n') for line in open('filename')]

Question

How do you call an external command (as if I’d typed it at the Unix shell or Windows command prompt) from within a Python script?

Answer accepted (score 4312)

Look at the subprocess module in the standard library:

import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess vs. system is that it is more flexible (you can get the stdout, stderr, the “real” status code, better error handling, etc…).

The official documentation recommends the subprocess module over the alternative os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function [os.system()].

The Replacing Older Functions with the subprocess Module section in the subprocess documentation may have some helpful recipes.

For versions of Python before 3.5, use call:

import subprocess
subprocess.call(["ls", "-l"])

Answer 2 (score 2852)

Here’s a summary of the ways to call external programs and the advantages and disadvantages of each:

1. os.system("some_command with args") passes the command and arguments to your system’s shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example:

   os.system("some_command < input_file | another_command > output_file")  
   ```</li>
   </ol>
   
   However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc.  On the other hand, this also lets you run commands which are simply shell commands and not actually external programs.  See <a href="https://docs.python.org/2/library/os.html#os.system" rel="noreferrer">the documentation</a>.  
   
   <ol start="2">
   <li><p>`stream = os.popen("some_command with args")` will do the same thing as `os.system` except that it gives you a file-like object that you can use to access standard input/output for that process.  There are 3 other variants of popen that all handle the i/o slightly differently.  If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything.  See <a href="https://docs.python.org/2/library/os.html#os.popen" rel="noreferrer">the documentation</a>.</p></li>
   <li><p>The `Popen` class of the `subprocess` module.  This is intended as a replacement for `os.popen` but has the downside of being slightly more complicated by virtue of being so comprehensive.  For example, you'd say:</p>
   
   ```python
   print subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE).stdout.read()

   instead of:

   print os.popen("echo Hello World").read()

   but it is nice to have all of the options there in one unified class instead of 4 different popen functions. See the documentation.

2. The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply waits until the command completes and gives you the return code. For example:

   return_code = subprocess.call("echo Hello World", shell=True)  

   See the documentation.

3. If you’re on Python 3.5 or later, you can use the new subprocess.run function, which is a lot like the above but even more flexible and returns a CompletedProcess object when the command finishes executing.

4. The os module also has all of the fork/exec/spawn functions that you’d have in a C program, but I don’t recommend using them directly.

The subprocess module should probably be what you use.

Finally please be aware that for all methods where you pass the final command to be executed by the shell as a string and you are responsible for escaping it. There are serious security implications if any part of the string that you pass can not be fully trusted. For example, if a user is entering some/any part of the string. If you are unsure, only use these methods with constants. To give you a hint of the implications consider this code:

print subprocess.Popen("echo %s " % user_input, stdout=PIPE).stdout.read()

and imagine that the user enters something “my mama didnt love me && rm -rf /” which could erase the whole filesystem.

Answer 3 (score 318)

Typical implementation:

import subprocess

p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
    print line,
retval = p.wait()

You are free to do what you want with the stdout data in the pipe. In fact, you can simply omit those parameters (stdout= and stderr=) and it’ll behave like os.system().

Question

I want to convert an integer to a string in Python. I am typecasting it in vain:

d = 15
d.str()

When I try to convert it to string, it’s showing an error like int doesn’t have any attribute called str.

Answer accepted (score 1966)

>>> str(10)
'10'
>>> int('10')
10

Links to the documentation:

• int()
• str()

Conversion to a string is done with the builtin str() function, which basically calls the __str__() method of its parameter.

Answer 2 (score 112)

Try this:

str(i)

Answer 3 (score 54)

There is not typecast and no type coercion in Python. You have to convert your variable in an explicit way.

To convert an object in string you use the str() function. It works with any object that has a method called __str__() defined. In fact

str(a)

is equivalent to

a.__str__()

The same if you want to convert something to int, float, etc.

Question

Is it possible to add a key to a Python dictionary after it has been created? It doesn’t seem to have an .add() method.

Answer accepted (score 3128)

d = {'key':'value'}
print(d)
# {'key': 'value'}
d['mynewkey'] = 'mynewvalue'
print(d)
# {'mynewkey': 'mynewvalue', 'key': 'value'}

Answer 2 (score 975)

To add multiple keys simultaneously:

>>> x = {1:2}
>>> print x
{1: 2}

>>> d = {3:4, 5:6, 7:8}
>>> x.update(d)
>>> print x
{1: 2, 3: 4, 5: 6, 7: 8}

For adding a single key, the accepted answer has less computational overhead.

Answer 3 (score 838)

I feel like consolidating info about Python dictionaries:

Creating an empty dictionary

data = {}
# OR
data = dict()

Creating a dictionary with initial values

data = {'a':1,'b':2,'c':3}
# OR
data = dict(a=1, b=2, c=3)
# OR
data = {k: v for k, v in (('a', 1),('b',2),('c',3))}

Inserting/Updating a single value

data['a']=1  # Updates if 'a' exists, else adds 'a'
# OR
data.update({'a':1})
# OR
data.update(dict(a=1))
# OR
data.update(a=1)

Inserting/Updating multiple values

data.update({'c':3,'d':4})  # Updates 'c' and adds 'd'

Creating a merged dictionary without modifying originals

data3 = {}
data3.update(data)  # Modifies data3, not data
data3.update(data2)  # Modifies data3, not data2

Deleting items in dictionary

del data[key]  # Removes specific element in a dictionary
data.pop(key)  # Removes the key & returns the value
data.clear()  # Clears entire dictionary

Check if a key is already in dictionary

key in data

Iterate through pairs in a dictionary

for key in data: # Iterates just through the keys, ignoring the values
for key, value in d.items(): # Iterates through the pairs
for key in d.keys(): # Iterates just through key, ignoring the values
for value in d.values(): # Iterates just through value, ignoring the keys

Create a dictionary from 2 lists

data = dict(zip(list_with_keys, list_with_values))

New to Python3

Creating a merged dictionary without modifying originals

data = {**data1, **data2, **data3}

Feel free to add more!

Question

I wanted to test if a key exists in a dictionary before updating the value for the key. I wrote the following code:

if 'key1' in dict.keys():
  print "blah"
else:
  print "boo"

I think this is not the best way to accomplish this task. Is there a better way to test for a key in the dictionary?

Answer accepted (score 2923)

in is the intended way to test for the existence of a key in a dict.

d = dict()

for i in range(100):
    key = i % 10
    if key in d:
        d[key] += 1
    else:
        d[key] = 1

If you wanted a default, you can always use dict.get():

d = dict()

for i in range(100):
    key = i % 10
    d[key] = d.get(key, 0) + 1

… and if you wanted to always ensure a default value for any key you can use defaultdict from the collections module, like so:

from collections import defaultdict

d = defaultdict(int)

for i in range(100):
    d[i % 10] += 1

… but in general, the in keyword is the best way to do it.

Answer 2 (score 1454)

You don’t have to call keys:

if 'key1' in dict:
  print "blah"
else:
  print "boo"

That will be much faster as it uses the dictionary’s hashing as opposed to doing a linear search, which calling keys would do.

Answer 3 (score 258)

You can test for the presence of a key in a dictionary, using the in keyword:

d = {'a': 1, 'b': 2}
'a' in d # <== evaluates to True
'c' in d # <== evaluates to False

A common use for checking the existence of a key in a dictionary before mutating it is to default-initialize the value (e.g. if your values are lists, for example, and you want to ensure that there is an empty list to which you can append when inserting the first value for a key). In cases such as those, you may find the collections.defaultdict() type to be of interest.

In older code, you may also find some uses of has_key(), a deprecated method for checking the existence of keys in dictionaries (just use key_name in dict_name, instead).

Question

Consider the following:

items = []
items.append("apple")
items.append("orange")
items.append("banana")

# FAKE METHOD:
items.amount()  # Should return 3

How do I get the number of elements in the list items?

Answer accepted (score 2526)

The len() function can be used with several different types in Python - both built-in types and library types. For example:

>>> len([1,2,3])
3

Official 2.x documentation is here: len()
Official 3.x documentation is here: len()

Answer 2 (score 216)

How to get the size of a list?

To find the size of a list, use the builtin function, len:

items = []
items.append("apple")
items.append("orange")
items.append("banana")

And now:

len(items)

returns 3.

Explanation

Everything in Python is an object, including lists. All objects have a header of some sort in the C implementation.

Lists and other similar builtin objects with a “size” in Python, in particular, have an attribute called ob_size, where the number of elements in the object is cached. So checking the number of objects in a list is very fast.

But if you’re checking if list size is zero or not, don’t use len - instead, put the list in a boolean context - it treated as False if empty, True otherwise.

From the docs

len(s)

Return the length (the number of items) of an object. The argument may be a sequence (such as a string, bytes, tuple, list, or range) or a collection (such as a dictionary, set, or frozen set).

len is implemented with __len__, from the data model docs:

object.__len__(self)

Called to implement the built-in function len(). Should return the length of the object, an integer >= 0. Also, an object that doesn’t define a __nonzero__() [in Python 2 or __bool__() in Python 3] method and whose __len__() method returns zero is considered to be false in a Boolean context.

And we can also see that __len__ is a method of lists:

items.__len__()

returns 3.

Builtin types you can get the len (length) of

And in fact we see we can get this information for all of the described types:

>>> all(hasattr(cls, '__len__') for cls in (str, bytes, tuple, list, 
                                            xrange, dict, set, frozenset))
True

Do not use len to test for an empty or nonempty list

To test for a specific length, of course, simply test for equality:

if len(items) == required_length:
    ...

But there’s a special case for testing for a zero length list or the inverse. In that case, do not test for equality.

Also, do not do:

if len(items): 
    ...

Instead, simply do:

if items:     # Then we have some items, not empty!
    ...

or

if not items: # Then we have an empty list!
    ...

I explain why here but in short, if items or if not items is both more readable and more performant.

Answer 3 (score 72)

While this may not be useful due to the fact that it’d make a lot more sense as being “out of the box” functionality, a fairly simple hack would be to build a class with a length property:

class slist(list):
    @property
    def length(self):
        return len(self)

You can use it like so:

>>> l = slist(range(10))
>>> l.length
10
>>> print l
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Essentially, it’s exactly identical to a list object, with the added benefit of having an OOP-friendly length property.

As always, your mileage may vary.

Question

How can I create or use a global variable in a function?

If I create a global variable in one function, how can I use that global variable in another function? Do I need to store the global variable in a local variable of the function which needs its access?

Answer accepted (score 4017)

You can use a global variable in other functions by declaring it as global in each function that assigns to it:

globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print(globvar)     # No need for global declaration to read value of globvar

set_globvar_to_one()
print_globvar()       # Prints 1

I imagine the reason for it is that, since global variables are so dangerous, Python wants to make sure that you really know that’s what you’re playing with by explicitly requiring the global keyword.

See other answers if you want to share a global variable across modules.

Answer 2 (score 736)

If I’m understanding your situation correctly, what you’re seeing is the result of how Python handles local (function) and global (module) namespaces.

Say you’ve got a module like this:

# sample.py
myGlobal = 5

def func1():
    myGlobal = 42

def func2():
    print myGlobal

func1()
func2()

You might expecting this to print 42, but instead it prints 5. As has already been mentioned, if you add a ‘global’ declaration to func1(), then func2() will print 42.

def func1():
    global myGlobal
    myGlobal = 42

What’s going on here is that Python assumes that any name that is assigned to, anywhere within a function, is local to that function unless explicitly told otherwise. If it is only reading from a name, and the name doesn’t exist locally, it will try to look up the name in any containing scopes (e.g. the module’s global scope).

When you assign 42 to the name myGlobal, therefore, Python creates a local variable that shadows the global variable of the same name. That local goes out of scope and is garbage-collected when func1() returns; meanwhile, func2() can never see anything other than the (unmodified) global name. Note that this namespace decision happens at compile time, not at runtime – if you were to read the value of myGlobal inside func1() before you assign to it, you’d get an UnboundLocalError, because Python has already decided that it must be a local variable but it has not had any value associated with it yet. But by using the ‘global’ statement, you tell Python that it should look elsewhere for the name instead of assigning to it locally.

(I believe that this behavior originated largely through an optimization of local namespaces – without this behavior, Python’s VM would need to perform at least three name lookups each time a new name is assigned to inside a function (to ensure that the name didn’t already exist at module/builtin level), which would significantly slow down a very common operation.)

Answer 3 (score 201)

You may want to explore the notion of namespaces. In Python, the module is the natural place for global data:

Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user’s global variables. On the other hand, if you know what you are doing you can touch a module’s global variables with the same notation used to refer to its functions, modname.itemname.

A specific use of global-in-a-module is described here - How do I share global variables across modules?, and for completeness the contents are shared here:

The canonical way to share information across modules within a single program is to create a special configuration module (often called config or cfg). Just import the configuration module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example:

File: config.py

x = 0   # Default value of the 'x' configuration setting

File: mod.py

import config
config.x = 1

File: main.py

import config
import mod
print config.x

Question

What is the module/method used to get the current time?

Answer accepted (score 2718)

Use:

>>> import datetime
>>> datetime.datetime.now()
datetime.datetime(2009, 1, 6, 15, 8, 24, 78915)

>>> print(datetime.datetime.now())
2018-07-29 09:17:13.812189

And just the time:

>>> datetime.datetime.now().time()
datetime.time(15, 8, 24, 78915)

>>> print(datetime.datetime.now().time())
09:17:51.914526

See the documentation for more information.

To save typing, you can import the datetime object from the datetime module:

>>> from datetime import datetime

Then remove the leading datetime. from all of the above.

Answer 2 (score 891)

You can use time.strftime():

>>> from time import gmtime, strftime
>>> strftime("%Y-%m-%d %H:%M:%S", gmtime())
'2009-01-05 22:14:39'

Answer 3 (score 531)

from datetime import datetime
datetime.now().strftime('%Y-%m-%d %H:%M:%S')

For this example, the output will be like this: '2013-09-18 11:16:32'

Here is the list of strftime directives.

Question

pip is a replacement for easy_install. But should I install pip using easy_install on Windows? Is there a better way?

Answer accepted (score 1779)

Python 2.7.9+ and 3.4+

Good news! Python 3.4 (released March 2014) and Python 2.7.9 (released December 2014) ship with Pip. This is the best feature of any Python release. It makes the community’s wealth of libraries accessible to everyone. Newbies are no longer excluded from using community libraries by the prohibitive difficulty of setup. In shipping with a package manager, Python joins Ruby, Node.js, Haskell, Perl, Go—almost every other contemporary language with a majority open-source community. Thank you, Python.

If you do find that pip is not available when using Python 3.4+ or Python 2.7.9+, simply execute e.g.:

py -3 -m ensurepip

Of course, that doesn’t mean Python packaging is problem solved. The experience remains frustrating. I discuss this in the Stack Overflow question Does Python have a package/module management system?.

And, alas for everyone using Python 2.7.8 or earlier (a sizable portion of the community). There’s no plan to ship Pip to you. Manual instructions follow.

Python 2 ≤ 2.7.8 and Python 3 ≤ 3.3

Flying in the face of its ‘batteries included’ motto, Python ships without a package manager. To make matters worse, Pip was—until recently—ironically difficult to install.

Official instructions

Per https://pip.pypa.io/en/stable/installing/#do-i-need-to-install-pip:

Download get-pip.py, being careful to save it as a .py file rather than .txt. Then, run it from the command prompt:

python get-pip.py

You possibly need an administrator command prompt to do this. Follow Start a Command Prompt as an Administrator (Microsoft TechNet).

This installs the pip package, which (in Windows) contains ….exe that path must be in PATH environment variable to use pip from the command line (see the second part of ‘Alternative Instructions’ for adding it to your PATH,

Alternative instructions

The official documentation tells users to install Pip and each of its dependencies from source. That’s tedious for the experienced and prohibitively difficult for newbies.

For our sake, Christoph Gohlke prepares Windows installers (.msi) for popular Python packages. He builds installers for all Python versions, both 32 and 64 bit. You need to:

1. Install setuptools
2. Install pip

For me, this installed Pip at C:\Python27\Scripts\pip.exe. Find pip.exe on your computer, then add its folder (for example, C:\Python27\Scripts) to your path (Start / Edit environment variables). Now you should be able to run pip from the command line. Try installing a package:

pip install httpie

There you go (hopefully)! Solutions for common problems are given below:

Proxy problems

If you work in an office, you might be behind an HTTP proxy. If so, set the environment variables http_proxy and https_proxy. Most Python applications (and other free software) respect these. Example syntax:

http://proxy_url:port
http://username:password@proxy_url:port

If you’re really unlucky, your proxy might be a Microsoft NTLM proxy. Free software can’t cope. The only solution is to install a free software friendly proxy that forwards to the nasty proxy. http://cntlm.sourceforge.net/

Unable to find vcvarsall.bat

Python modules can be partly written in C or C++. Pip tries to compile from source. If you don’t have a C/C++ compiler installed and configured, you’ll see this cryptic error message.

Error: Unable to find vcvarsall.bat

You can fix that by installing a C++ compiler such as MinGW or Visual C++. Microsoft actually ships one specifically for use with Python. Or try Microsoft Visual C++ Compiler for Python 2.7.

Often though it’s easier to check Christoph’s site for your package.

Answer 2 (score 296)

– Outdated – use distribute, not setuptools as described here. –
– Outdated #2 – use setuptools as distribute is deprecated.

As you mentioned pip doesn’t include an independent installer, but you can install it with its predecessor easy_install.

So:

1. Download the last pip version from here: http://pypi.python.org/pypi/pip#downloads
2. Uncompress it
3. Download the last easy installer for Windows: (download the .exe at the bottom of http://pypi.python.org/pypi/setuptools ). Install it.
4. copy the uncompressed pip folder content into C:\Python2x\ folder (don’t copy the whole folder into it, just the content), because python command doesn’t work outside C:\Python2x folder and then run: python setup.py install
5. Add your python C:\Python2x\Scripts to the path

You are done.

Now you can use pip install package to easily install packages as in Linux :)

Answer 3 (score 215)

2014 UPDATE:

1. If you have installed Python 3.4 or later, pip is included with Python and should already be working on your system.

2. If you are running a version below Python 3.4 or if pip was not installed with Python 3.4 for some reason, then you’d probably use pip’s official installation script get-pip.py. The pip installer now grabs setuptools for you, and works regardless of architecture (32-bit or 64-bit).

The installation instructions are detailed here and involve:

To install or upgrade pip, securely download get-pip.py.

Then run the following (which may require administrator access):

python get-pip.py

To upgrade an existing setuptools (or distribute), run pip install -U setuptools

I’ll leave the two sets of old instructions below for posterity.

OLD Answers:

For Windows editions of the 64 bit variety - 64-bit Windows + Python used to require a separate installation method due to ez_setup, but I’ve tested the new distribute method on 64-bit Windows running 32-bit Python and 64-bit Python, and you can now use the same method for all versions of Windows/Python 2.7X:

OLD Method 2 using distribute:

1. Download distribute - I threw mine in C:\Python27\Scripts (feel free to create a Scripts directory if it doesn’t exist.
2. Open up a command prompt (on Windows you should check out conemu2 if you don’t use PowerShell) and change (cd) to the directory you’ve downloaded distribute_setup.py to.
3. Run distribute_setup: python distribute_setup.py (This will not work if your python installation directory is not added to your path - go here for help)
4. Change the current directory to the Scripts directory for your Python installation (C:\Python27\Scripts) or add that directory, as well as the Python base installation directory to your %PATH% environment variable.
5. Install pip using the newly installed setuptools: easy_install pip

The last step will not work unless you’re either in the directory easy_install.exe is located in (C:would be the default for Python 2.7), or you have that directory added to your path.

OLD Method 1 using ez_setup:

from the setuptools page –

Download ez_setup.py and run it; it will download the appropriate .egg file and install it for you. (Currently, the provided .exe installer does not support 64-bit versions of Python for Windows, due to a distutils installer compatibility issue.

After this, you may continue with:

2. Add c:\Python2x\Scripts to the Windows path (replace the x in Python2x with the actual version number you have installed)
3. Open a new (!) DOS prompt. From there run easy_install pip

Question

I would like to know how to put a time delay in a Python script.

Answer 2 (score 2835)

import time
time.sleep(5)   # Delays for 5 seconds. You can also use a float value.

Here is another example where something is run approximately once a minute:

import time
while True:
    print("This prints once a minute.")
    time.sleep(60) # Delay for 1 minute (60 seconds).

Answer 3 (score 736)

You can use the sleep() function in the time module. It can take a float argument for sub-second resolution.

from time import sleep
sleep(0.1) # Time in seconds

Question

What’s the difference between the list methods append() and extend()?

Answer accepted (score 4958)

append: Appends object at the end.

x = [1, 2, 3]
x.append([4, 5])
print (x)

gives you: [1, 2, 3, [4, 5]]

extend: Extends list by appending elements from the iterable.

x = [1, 2, 3]
x.extend([4, 5])
print (x)

gives you: [1, 2, 3, 4, 5]

Answer 2 (score 606)

append adds an element to a list, and extend concatenates the first list with another list (or another iterable, not necessarily a list.)

>>> li = ['a', 'b', 'mpilgrim', 'z', 'example']
>>> li
['a', 'b', 'mpilgrim', 'z', 'example']

>>> li.append("new")
>>> li
['a', 'b', 'mpilgrim', 'z', 'example', 'new']

>>> li.append(["new", 2])
>>> li
['a', 'b', 'mpilgrim', 'z', 'example', 'new', ['new', 2]]

>>> li.insert(2, "new")
>>> li
['a', 'b', 'new', 'mpilgrim', 'z', 'example', 'new', ['new', 2]]

>>> li.extend(["two", "elements"])
>>> li
['a', 'b', 'new', 'mpilgrim', 'z', 'example', 'new', ['new', 2], 'two', 'elements']

Answer 3 (score 437)

What is the difference between the list methods append and extend?

• append adds its argument as a single element to the end of a list. The length of the list itself will increase by one.
• extend iterates over its argument adding each element to the list, extending the list. The length of the list will increase by however many elements were in the iterable argument.

append

The list.append method appends an object to the end of the list.

my_list.append(object) 

Whatever the object is, whether a number, a string, another list, or something else, it gets added onto the end of my_list as a single entry on the list.

>>> my_list
['foo', 'bar']
>>> my_list.append('baz')
>>> my_list
['foo', 'bar', 'baz']

So keep in mind that a list is an object. If you append another list onto a list, the first list will be a single object at the end of the list (which may not be what you want):

>>> another_list = [1, 2, 3]
>>> my_list.append(another_list)
>>> my_list
['foo', 'bar', 'baz', [1, 2, 3]]
                     #^^^^^^^^^--- single item at the end of the list.

extend

The list.extend method extends a list by appending elements from an iterable:

my_list.extend(iterable)

So with extend, each element of the iterable gets appended onto the list. For example:

>>> my_list
['foo', 'bar']
>>> another_list = [1, 2, 3]
>>> my_list.extend(another_list)
>>> my_list
['foo', 'bar', 1, 2, 3]

Keep in mind that a string is an iterable, so if you extend a list with a string, you’ll append each character as you iterate over the string (which may not be what you want):

>>> my_list.extend('baz')
>>> my_list
['foo', 'bar', 1, 2, 3, 'b', 'a', 'z']

Operator Overload, __add__ (+) and __iadd__ (+=)

Both + and += operators are defined for list. They are semantically similar to extend.

my_list + another_list creates a third list in memory, so you can return the result of it, but it requires that the second iterable be a list.

my_list += another_list modifies the list in-place (it is the in-place operator, and lists are mutable objects, as we’ve seen) so it does not create a new list. It also works like extend, in that the second iterable can be any kind of iterable.

Don’t get confused - my_list = my_list + another_list is not equivalent to += - it gives you a brand new list assigned to my_list.

Time Complexity

Append has constant time complexity, O(1).

Extend has time complexity, O(k).

Iterating through the multiple calls to append adds to the complexity, making it equivalent to that of extend, and since extend’s iteration is implemented in C, it will always be faster if you intend to append successive items from an iterable onto a list.

Performance

You may wonder what is more performant, since append can be used to achieve the same outcome as extend. The following functions do the same thing:

def append(alist, iterable):
    for item in iterable:
        alist.append(item)

def extend(alist, iterable):
    alist.extend(iterable)

So let’s time them:

import timeit

>>> min(timeit.repeat(lambda: append([], "abcdefghijklmnopqrstuvwxyz")))
2.867846965789795
>>> min(timeit.repeat(lambda: extend([], "abcdefghijklmnopqrstuvwxyz")))
0.8060121536254883

Addressing a comment on timings

A commenter said:

Perfect answer, I just miss the timing of comparing adding only one element

Do the semantically correct thing. If you want to append all elements in an iterable, use extend. If you’re just adding one element, use append.

Ok, so let’s create an experiment to see how this works out in time:

def append_one(a_list, element):
    a_list.append(element)

def extend_one(a_list, element):
    """creating a new list is semantically the most direct
    way to create an iterable to give to extend"""
    a_list.extend([element])

import timeit

And we see that going out of our way to create an iterable just to use extend is a (minor) waste of time:

>>> min(timeit.repeat(lambda: append_one([], 0)))
0.2082819009956438
>>> min(timeit.repeat(lambda: extend_one([], 0)))
0.2397019260097295

We learn from this that there’s nothing gained from using extend when we have only one element to append.

Also, these timings are not that important. I am just showing them to make the point that, in Python, doing the semantically correct thing is doing things the Right Way™.

It’s conceivable that you might test timings on two comparable operations and get an ambiguous or inverse result. Just focus on doing the semantically correct thing.

Conclusion

We see that extend is semantically clearer, and that it can run much faster than append, when you intend to append each element in an iterable to a list.

If you only have a single element (not in an iterable) to add to the list, use append.

Question

I want a to be rounded to 13.95.

>>> a
13.949999999999999
>>> round(a, 2)
13.949999999999999

The round function does not work the way I expected.

Answer 2 (score 1484)

You are running into the old problem with floating point numbers that not all numbers can be represented exactly. The command line is just showing you the full floating point form from memory.

With floating point representation, your rounded version is the same number. Since computers are binary, they store floating point numbers as an integer and then divide it by a power of two so 13.95 will be represented in a similar fashion to 125650429603636838/(2**53).

Double precision numbers have 53 bits (16 digits) of precision and regular floats have 24 bits (8 digits) of precision. The floating point type in Python uses double precision to store the values.

For example,

  >>> 125650429603636838/(2**53)
  13.949999999999999

  >>> 234042163/(2**24)
  13.949999988079071

  >>> a=13.946
  >>> print(a)
  13.946
  >>> print("%.2f" % a)
  13.95
  >>> round(a,2)
  13.949999999999999
  >>> print("%.2f" % round(a,2))
  13.95
  >>> print("{0:.2f}".format(a))
  13.95
  >>> print("{0:.2f}".format(round(a,2)))
  13.95
  >>> print("{0:.15f}".format(round(a,2)))
  13.949999999999999

If you are after only two decimal places (to display a currency value, for example), then you have a couple of better choices:

1. Use integers and store values in cents, not dollars and then divide by 100 to convert to dollars.
2. Or use a fixed point number like decimal.

Answer 3 (score 525)

There are new format specifications, String Format Specification Mini-Language:

You can do the same as:

"{0:.2f}".format(13.949999999999999)

Note that the above returns a string. In order to get as float, simply wrap with float(...):

float("{0:.2f}".format(13.949999999999999))

Note that wrapping with float() doesn’t change anything:

>>> x = 13.949999999999999999
>>> x
13.95
>>> g = float("{0:.2f}".format(x))
>>> g
13.95
>>> x == g
True
>>> h = round(x, 2)
>>> h
13.95
>>> x == h
True

Question

I’ve got a huge list of date-times like this as strings:

Jun 1 2005  1:33PM
Aug 28 1999 12:00AM

I’m going to be shoving these back into proper datetime fields in a database so I need to magic them into real datetime objects.

This is going through Django’s ORM so I can’t use SQL to do the conversion on insert.

Answer accepted (score 3174)

datetime.strptime is the main routine for parsing strings into datetimes. It can handle all sorts of formats, with the format determined by a format string you give it:

from datetime import datetime

datetime_object = datetime.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')

The resulting datetime object is timezone-naive.

Links:

• Python documentation for strptime: Python 2, Python 3

• Python documentation for strptime/strftime format strings: Python 2, Python 3

• strftime.org is also a really nice reference for strftime

Notes:

• strptime = “string parse time”
• strftime = “string format time”
• Pronounce it out loud today & you won’t have to search for it again in 6 months.

Answer 2 (score 757)

Use the third party dateutil library:

from dateutil import parser
parser.parse("Aug 28 1999 12:00AM")  # datetime.datetime(1999, 8, 28, 0, 0)

It can handle most date formats, including the one you need to parse. It’s more convenient than strptime as it can guess the correct format most of the time.

It’s very useful for writing tests, where readability is more important than performance.

You can install it with:

pip install python-dateutil

Answer 3 (score 477)

Check out strptime in the time module. It is the inverse of strftime.

$ python
>>> import time
>>> time.strptime('Jun 1 2005  1:33PM', '%b %d %Y %I:%M%p')
time.struct_time(tm_year=2005, tm_mon=6, tm_mday=1,
                 tm_hour=13, tm_min=33, tm_sec=0,
                 tm_wday=2, tm_yday=152, tm_isdst=-1)

Question

Is there a way to substring a string in Python, to get a new string from the third character to the end of the string?

Maybe like myString[2:end]?

If leaving the second part means ‘till the end’, and if you leave the first part, does it start from the start?

Answer accepted (score 2939)

>>> x = "Hello World!"
>>> x[2:]
'llo World!'
>>> x[:2]
'He'
>>> x[:-2]
'Hello Worl'
>>> x[-2:]
'd!'
>>> x[2:-2]
'llo Worl'

Python calls this concept “slicing” and it works on more than just strings. Take a look here for a comprehensive introduction.

Answer 2 (score 361)

Just for completeness as nobody else has mentioned it. The third parameter to an array slice is a step. So reversing a string is as simple as:

some_string[::-1]

Or selecting alternate characters would be:

"H-e-l-l-o- -W-o-r-l-d"[::2] # outputs "Hello World"

The ability to step forwards and backwards through the string maintains consistency with being able to array slice from the start or end.

Answer 3 (score 108)

Substr() normally (i.e. PHP and Perl) works this way:

s = Substr(s, beginning, LENGTH)

So the parameters are beginning and LENGTH.

But Python’s behaviour is different; it expects beginning and one after END (!). This is difficult to spot by beginners. So the correct replacement for Substr(s, beginning, LENGTH) is

s = s[ beginning : beginning + LENGTH]

Question

In Python, what commands can I use to find:

1. the current directory (where I was in the terminal when I ran the Python script), and
2. where the file I am executing is?

Answer accepted (score 3058)

To get the full path to the directory a Python file is contained in, write this in that file:

import os 
dir_path = os.path.dirname(os.path.realpath(__file__))

(Note that the incantation above won’t work if you’ve already used os.chdir() to change your current working directory, since the value of the __file__ constant is relative to the current working directory and is not changed by an os.chdir() call.)

To get the current working directory use

import os
cwd = os.getcwd()

Documentation references for the modules, constants and functions used above:

• The os and os.path modules.
• The __file__ constant
• os.path.realpath(path) (returns “the canonical path of the specified filename, eliminating any symbolic links encountered in the path”)
• os.path.dirname(path) (returns “the directory name of pathname path”)
• os.getcwd() (returns “a string representing the current working directory”)
• os.chdir(path) (“change the current working directory to path”)

Answer 2 (score 308)

Current Working Directory: os.getcwd()

And the __file__ attribute can help you find out where the file you are executing is located. This SO post explains everything: How do I get the path of the current executed file in Python?

Answer 3 (score 260)

You may find this useful as a reference:

import os

print("Path at terminal when executing this file")
print(os.getcwd() + "\n")

print("This file path, relative to os.getcwd()")
print(__file__ + "\n")

print("This file full path (following symlinks)")
full_path = os.path.realpath(__file__)
print(full_path + "\n")

print("This file directory and name")
path, filename = os.path.split(full_path)
print(path + ' --> ' + filename + "\n")

print("This file directory only")
print(os.path.dirname(full_path))

Question

I have this JSON in a file:

{
    "maps": [
        {
            "id": "blabla",
            "iscategorical": "0"
        },
        {
            "id": "blabla",
            "iscategorical": "0"
        }
    ],
    "masks": [
        "id": "valore"
    ],
    "om_points": "value",
    "parameters": [
        "id": "valore"
    ]
}

I wrote this script to print all of the JSON data:

import json
from pprint import pprint

with open('data.json') as f:
    data = json.load(f)

pprint(data)

This program raises an exception, though:

Traceback (most recent call last):
  File "<pyshell#1>", line 5, in <module>
    data = json.load(f)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 13 column 13 (char 213)

How can I parse the JSON and extract its values?

Answer accepted (score 2081)

Your data is not valid JSON format. You have [] when you should have {}:

• [] are for JSON arrays, which are called list in Python
• {} are for JSON objects, which are called dict in Python

Here’s how your JSON file should look:

{
    "maps": [
        {
            "id": "blabla",
            "iscategorical": "0"
        },
        {
            "id": "blabla",
            "iscategorical": "0"
        }
    ],
    "masks": {
        "id": "valore"
    },
    "om_points": "value",
    "parameters": {
        "id": "valore"
    }
}

Then you can use your code:

import json
from pprint import pprint

with open('data.json') as f:
    data = json.load(f)

pprint(data)

With data, you can now also find values like so:

data["maps"][0]["id"]
data["masks"]["id"]
data["om_points"]

Try those out and see if it starts to make sense.

Answer 2 (score 304)

Your data.json should look like this:

{
 "maps":[
         {"id":"blabla","iscategorical":"0"},
         {"id":"blabla","iscategorical":"0"}
        ],
"masks":
         {"id":"valore"},
"om_points":"value",
"parameters":
         {"id":"valore"}
}

Your code should be:

import json
from pprint import pprint

with open('data.json') as data_file:    
    data = json.load(data_file)
pprint(data)

Note that this only works in Python 2.6 and up, as it depends upon the with-statement. In Python 2.5 use from __future__ import with_statement, in Python <= 2.4, see Justin Peel’s answer, which this answer is based upon.

You can now also access single values like this:

data["maps"][0]["id"]  # will return 'blabla'
data["masks"]["id"]    # will return 'valore'
data["om_points"]      # will return 'value'

Answer 3 (score 64)

Justin Peel’s answer is really helpful, but if you are using Python 3 reading JSON should be done like this:

with open('data.json', encoding='utf-8') as data_file:
    data = json.loads(data_file.read())

Note: use json.loads instead of json.load. In Python 3, json.loads takes a string parameter. json.load takes a file-like object parameter. data_file.read() returns a string object.

To be honest, I don’t think it’s a problem to load all json data into memory most cases.

Question

What does the if __name__ == "__main__": do?

# Threading example
import time, thread

def myfunction(string, sleeptime, lock, *args):
    while True:
        lock.acquire()
        time.sleep(sleeptime)
        lock.release()
        time.sleep(sleeptime)

if __name__ == "__main__":
    lock = thread.allocate_lock()
    thread.start_new_thread(myfunction, ("Thread #: 1", 2, lock))
    thread.start_new_thread(myfunction, ("Thread #: 2", 2, lock))

Answer accepted (score 5891)

Whenever the Python interpreter reads a source file, it does two things:

• it sets a few special variables like __name__, and then

• it executes all of the code found in the file.

Let’s see how this works and how it relates to your question about the __name__ checks we always see in Python scripts.

Code Sample

Let’s use a slightly different code sample to explore how imports and scripts work. Suppose the following is in a file called foo.py.

# Suppose this is foo.py.

print("before import")
import math

print("before functionA")
def functionA():
    print("Function A")

print("before functionB")
def functionB():
    print("Function B {}".format(math.sqrt(100)))

print("before __name__ guard")
if __name__ == '__main__':
    functionA()
    functionB()
print("after __name__ guard")

Special Variables

When the Python interpeter reads a source file, it first defines a few special variables. In this case, we care about the __name__ variable.

When Your Module Is the Main Program

If you are running your module (the source file) as the main program, e.g. 

python foo.py

the interpreter will assign the hard-coded string "__main__" to the __name__ variable, i.e. 

# It's as if the interpreter inserts this at the top
# of your module when run as the main program.
__name__ = "__main__" 

When Your Module Is Imported By Another

On the other hand, suppose some other module is the main program and it imports your module. This means there’s a statement like this in the main program, or in some other module the main program imports:

# Suppose this is in some other main program.
import foo

In this case, the interpreter will look at the filename of your module, foo.py, strip off the .py, and assign that string to your module’s __name__ variable, i.e. 

# It's as if the interpreter inserts this at the top
# of your module when it's imported from another module.
__name__ = "foo"

Executing the Module’s Code

After the special variables are set up, the interpreter executes all the code in the module, one statement at a time. You may want to open another window on the side with the code sample so you can follow along with this explanation.

Always

1. It prints the string "before import" (without quotes).

2. It loads the math module and assigns it to a variable called math. This is equivalent to replacing import math with the following (note that __import__ is a low-level function in Python that takes a string and triggers the actual import):

# Find and load a module given its string name, "math",
# then assign it to a local variable called math.
math = __import__("math")

3. It prints the string "before functionA".

4. It executes the def block, creating a function object, then assigning that function object to a variable called functionA.

5. It prints the string "before functionB".

6. It executes the second def block, creating another function object, then assigning it to a variable called functionB.

7. It prints the string "before __name__ guard".

Only When Your Module Is the Main Program

8. If your module is the main program, then it will see that __name__ was indeed set to "__main__" and it calls the two functions, printing the strings "Function A" and "Function B 10.0".

Only When Your Module Is Imported by Another

8. (instead) If your module is not the main program but was imported by another one, then __name__ will be "foo", not "__main__", and it’ll skip the body of the if statement.

Always

9. It will print the string "after __name__ guard" in both situations.

Summary

In summary, here’s what’d be printed in the two cases:

# What gets printed if foo is the main program
before import
before functionA
before functionB
before __name__ guard
Function A
Function B 10.0
after __name__ guard

# What gets printed if foo is imported as a regular module
before import
before functionA
before functionB
before __name__ guard
after __name__ guard

Why Does It Work This Way?

You might naturally wonder why anybody would want this. Well, sometimes you want to write a .py file that can be both used by other programs and/or modules as a module, and can also be run as the main program itself. Examples:

• Your module is a library, but you want to have a script mode where it runs some unit tests or a demo.

• Your module is only used as a main program, but it has some unit tests, and the testing framework works by importing .py files like your script and running special test functions. You don’t want it to try running the script just because it’s importing the module.

• Your module is mostly used as a main program, but it also provides a programmer-friendly API for advanced users.

Beyond those examples, it’s elegant that running a script in Python is just setting up a few magic variables and importing the script. “Running” the script is a side effect of importing the script’s module.

Food for Thought

• Question: Can I have multiple __name__ checking blocks? Answer: it’s strange to do so, but the language won’t stop you.

• Suppose the following is in foo2.py. What happens if you say python foo2.py on the command-line? Why?

# Suppose this is foo2.py.

def functionA():
    print("a1")
    from foo2 import functionB
    print("a2")
    functionB()
    print("a3")

def functionB():
    print("b")

print("t1")
if __name__ == "__main__":
    print("m1")
    functionA()
    print("m2")
print("t2")

• Now, figure out what will happen if you remove the __name__ check in foo3.py:

# Suppose this is foo3.py.

def functionA():
    print("a1")
    from foo3 import functionB
    print("a2")
    functionB()
    print("a3")

def functionB():
    print("b")

print("t1")
print("m1")
functionA()
print("m2")
print("t2")

• What will this do when used as a script? When imported as a module?

# Suppose this is in foo4.py
__name__ = "__main__"

def bar():
    print("bar")

print("before __name__ guard")
if __name__ == "__main__":
    bar()
print("after __name__ guard")

Answer 2 (score 1696)

When your script is run by passing it as a command to the Python interpreter,

python myscript.py

all of the code that is at indentation level 0 gets executed. Functions and classes that are defined are, well, defined, but none of their code gets run. Unlike other languages, there’s no main() function that gets run automatically - the main() function is implicitly all the code at the top level.

In this case, the top-level code is an if block. __name__ is a built-in variable which evaluates to the name of the current module. However, if a module is being run directly (as in myscript.py above), then __name__ instead is set to the string "__main__". Thus, you can test whether your script is being run directly or being imported by something else by testing

if __name__ == "__main__":
    ...

If your script is being imported into another module, its various function and class definitions will be imported and its top-level code will be executed, but the code in the then-body of the if clause above won’t get run as the condition is not met. As a basic example, consider the following two scripts:

# file one.py
def func():
    print("func() in one.py")

print("top-level in one.py")

if __name__ == "__main__":
    print("one.py is being run directly")
else:
    print("one.py is being imported into another module")

# file two.py
import one

print("top-level in two.py")
one.func()

if __name__ == "__main__":
    print("two.py is being run directly")
else:
    print("two.py is being imported into another module")

Now, if you invoke the interpreter as

python one.py

The output will be

top-level in one.py
one.py is being run directly

If you run two.py instead:

python two.py

You get

top-level in one.py
one.py is being imported into another module
top-level in two.py
func() in one.py
two.py is being run directly

Thus, when module one gets loaded, its __name__ equals "one" instead of "__main__".

Answer 3 (score 676)

The simplest explanation for the __name__ variable (imho) is the following:

Create the following files.

# a.py
import b

and

# b.py
print "Hello World from %s!" % __name__

if __name__ == '__main__':
    print "Hello World again from %s!" % __name__

Running them will get you this output:

$ python a.py
Hello World from b!

As you can see, when a module is imported, Python sets globals()['__name__'] in this module to the module’s name. Also, upon import all the code in the module is being run. As the if statement evaluates to False this part is not executed.

$ python b.py
Hello World from __main__!
Hello World again from __main__!

As you can see, when a file is executed, Python sets globals()['__name__'] in this file to "__main__". This time, the if statement evaluates to True and is being run.

Question

For example, if passed the following:

a = []

How do I check to see if a is empty?

Answer accepted (score 4944)

if not a:
  print("List is empty")

Using the implicit booleanness of the empty list is quite pythonic.

Answer 2 (score 1081)

The pythonic way to do it is from the PEP 8 style guide (where Yes means “recommended” and No means “not recommended”):

For sequences, (strings, lists, tuples), use the fact that empty sequences are false.

<b>Yes:</b> if not seq:
     if seq:

<b>No:</b>  if len(seq):
     if not len(seq):

Answer 3 (score 693)

I prefer it explicitly:

if len(li) == 0:
    print('the list is empty')

This way it’s 100% clear that li is a sequence (list) and we want to test its size. My problem with if not li: ... is that it gives the false impression that li is a boolean variable.

Question

I have a dictionary of values read from two fields in a database: a string field and a numeric field. The string field is unique, so that is the key of the dictionary.

I can sort on the keys, but how can I sort based on the values?

Note: I have read Stack Overflow question here How do I sort a list of dictionaries by a value of the dictionary? and probably could change my code to have a list of dictionaries, but since I do not really need a list of dictionaries I wanted to know if there is a simpler solution to sort either in ascending or descending order.

Answer accepted (score 4391)

It is not possible to sort a dictionary, only to get a representation of a dictionary that is sorted. Dictionaries are inherently orderless, but other types, such as lists and tuples, are not. So you need an ordered data type to represent sorted values, which will be a list—probably a list of tuples.

For instance,

import operator
x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(x.items(), key=operator.itemgetter(1))

sorted_x will be a list of tuples sorted by the second element in each tuple. dict(sorted_x) == x.

And for those wishing to sort on keys instead of values:

import operator
x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(x.items(), key=operator.itemgetter(0))

In Python3 since unpacking is not allowed [1] we can use

x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(x.items(), key=lambda kv: kv[1])

If you want the output as a dict, you can use collections.OrderedDict:

import collections

sorted_dict = collections.OrderedDict(sorted_x)

Answer 2 (score 1136)

As simple as: sorted(dict1, key=dict1.get)

Well, it is actually possible to do a “sort by dictionary values”. Recently I had to do that in a Code Golf (Stack Overflow question Code golf: Word frequency chart). Abridged, the problem was of the kind: given a text, count how often each word is encountered and display a list of the top words, sorted by decreasing frequency.

If you construct a dictionary with the words as keys and the number of occurrences of each word as value, simplified here as:

from collections import defaultdict
d = defaultdict(int)
for w in text.split():
  d[w] += 1

then you can get a list of the words, ordered by frequency of use with sorted(d, key=d.get) - the sort iterates over the dictionary keys, using the number of word occurrences as a sort key .

for w in sorted(d, key=d.get, reverse=True):
  print w, d[w]

I am writing this detailed explanation to illustrate what people often mean by “I can easily sort a dictionary by key, but how do I sort by value” - and I think the OP was trying to address such an issue. And the solution is to do sort of list of the keys, based on the values, as shown above.

Answer 3 (score 750)

You could use:

sorted(d.items(), key=lambda x: x[1])

This will sort the dictionary by the values of each entry within the dictionary from smallest to largest.

To sort it in descending order just add reverse=True:

sorted(d.items(), key=lambda x: x[1], reverse=True)

Question

What is the most elegant way to check if the directory a file is going to be written to exists, and if not, create the directory using Python? Here is what I tried:

import os

file_path = "/my/directory/filename.txt"
directory = os.path.dirname(file_path)

try:
    os.stat(directory)
except:
    os.mkdir(directory)       

f = file(filename)

Somehow, I missed os.path.exists (thanks kanja, Blair, and Douglas). This is what I have now:

def ensure_dir(file_path):
    directory = os.path.dirname(file_path)
    if not os.path.exists(directory):
        os.makedirs(directory)

Is there a flag for “open”, that makes this happen automatically?

Answer accepted (score 4670)

I see two answers with good qualities, each with a small flaw, so I will give my take on it:

Try os.path.exists, and consider os.makedirs for the creation.

import os
if not os.path.exists(directory):
    os.makedirs(directory)

As noted in comments and elsewhere, there’s a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.

One option would be to trap the OSError and examine the embedded error code (see Is there a cross-platform way of getting information from Python’s OSError):

import os, errno

try:
    os.makedirs(directory)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise

Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one – we could still be fooled.

Depending on the application, the danger of concurrent operations may be more or less than the danger posed by other factors such as file permissions. The developer would have to know more about the particular application being developed and its expected environment before choosing an implementation.

Modern versions of Python improve this code quite a bit, both by exposing FileExistsError (in 3.3+)…

try:
    os.makedirs("path/to/directory")
except FileExistsError:
    # directory already exists
    pass

…and by allowing a keyword argument to os.makedirs called exist_ok (in 3.2+).

os.makedirs("path/to/directory", exist_ok=True)  # succeeds even if directory exists.

Answer 2 (score 1155)

Python 3.5+:

import pathlib
pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True) 

pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don’t need or want the parents to be created, skip the parents argument.

Python 3.2+:

Using pathlib:

If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.

If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.

Using os:

import os
os.makedirs(path, exist_ok=True)

os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.

Python 2.7+:

Using pathlib:

If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.

Using os:

import os
try: 
    os.makedirs(path)
except OSError:
    if not os.path.isdir(path):
        raise

While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.

Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.

Alternative:

mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3.

import distutils.dir_util
distutils.dir_util.mkpath(path)

Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn’t rely on any such cache. This limitation may be okay for some applications.

With regard to the directory’s mode, please refer to the documentation if you care about it.

Answer 3 (score 600)

Using try except and the right error code from errno module gets rid of the race condition and is cross-platform:

import os
import errno

def make_sure_path_exists(path):
    try:
        os.makedirs(path)
    except OSError as exception:
        if exception.errno != errno.EEXIST:
            raise

In other words, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir ‘a’ beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).

Question

How do I remove an element from a list by index in Python?

I found the list.remove method, but say I want to remove the last element, how do I do this? It seems like the default remove searches the list, but I don’t want any search to be performed.

Answer accepted (score 1546)

Use del and specify the index of the element you want to delete:

>>> a = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> del a[-1]
>>> a
[0, 1, 2, 3, 4, 5, 6, 7, 8]

Also supports slices:

>>> del a[2:4]
>>> a
[0, 1, 4, 5, 6, 7, 8, 9]

Here is the section from the tutorial.

Answer 2 (score 595)

You probably want pop:

a = ['a', 'b', 'c', 'd']
a.pop(1)

# now a is ['a', 'c', 'd']

By default, pop without any arguments removes the last item:

a = ['a', 'b', 'c', 'd']
a.pop()

# now a is ['a', 'b', 'c']

Answer 3 (score 124)

Like others mentioned pop and del are the efficient ways to remove an item of given index. Yet just for the sake of completion (since the same thing can be done via many ways in Python):

Using slices (this does not do in place removal of item from original list):

(Also this will be the least efficient method when working with Python list, but this could be useful (but not efficient, I reiterate) when working with user defined objects that do not support pop, yet do define a __getitem__ ):

>>> a = [1, 2, 3, 4, 5, 6]
>>> index = 3 # Only positive index

>>> a = a[:index] + a[index+1 :]
# a is now [1, 2, 3, 5, 6]

Note: Please note that this method does not modify the list in place like pop and del. It instead makes two copies of lists (one from the start until the index but without it (a[:index]) and one after the index till the last element (a[index+1:])) and creates a new list object by adding both. This is then reassigned to the list variable (a). The old list object is hence dereferenced and hence garbage collected (provided the original list object is not referenced by any variable other than a).

This makes this method very inefficient and it can also produce undesirable side effects (especially when other variables point to the original list object which remains un-modified).

Thanks to @MarkDickinson for pointing this out …

This Stack Overflow answer explains the concept of slicing.

Also note that this works only with positive indices.

While using with objects, the __getitem__ method must have been defined and more importantly the __add__ method must have been defined to return an object containing items from both the operands.

In essence, this works with any object whose class definition is like:

class foo(object):
    def __init__(self, items):
        self.items = items

    def __getitem__(self, index):
        return foo(self.items[index])

    def __add__(self, right):
        return foo( self.items + right.items )

This works with list which defines __getitem__ and __add__ methods.

Comparison of the three ways in terms of efficiency:

Assume the following is predefined:

a = range(10)
index = 3

The del object[index] method:

By far the most efficient method. It works will all objects that define a __del__ method.

The disassembly is as follows:

Code:

def del_method():
    global a
    global index
    del a[index]

Disassembly:

 10    0 LOAD_GLOBAL     0 (a)
       3 LOAD_GLOBAL     1 (index)
       6 DELETE_SUBSCR   # This is the line that deletes the item
       7 LOAD_CONST      0 (None)
      10 RETURN_VALUE
None

pop method:

It is less efficient than the del method and is used when you need to get the deleted item.

Code:

def pop_method():
    global a
    global index
    a.pop(index)

Disassembly:

 17     0 LOAD_GLOBAL     0 (a)
        3 LOAD_ATTR       1 (pop)
        6 LOAD_GLOBAL     2 (index)
        9 CALL_FUNCTION   1
       12 POP_TOP
       13 LOAD_CONST      0 (None)
       16 RETURN_VALUE

The slice and add method.

The least efficient.

Code:

def slice_method():
    global a
    global index
    a = a[:index] + a[index+1:]

Disassembly:

 24     0 LOAD_GLOBAL    0 (a)
        3 LOAD_GLOBAL    1 (index)
        6 SLICE+2
        7 LOAD_GLOBAL    0 (a)
       10 LOAD_GLOBAL    1 (index)
       13 LOAD_CONST     1 (1)
       16 BINARY_ADD
       17 SLICE+1
       18 BINARY_ADD
       19 STORE_GLOBAL   0 (a)
       22 LOAD_CONST     0 (None)
       25 RETURN_VALUE
None

Note: In all three disassembles ignore the last two lines which basically are return None. Also the first two lines are loading the global values a and index.

Question

I have a DataFrame using pandas and column labels that I need to edit to replace the original column labels.

I’d like to change the column names in a DataFrame A where the original column names are:

['$a', '$b', '$c', '$d', '$e'] 

to

['a', 'b', 'c', 'd', 'e'].

I have the edited column names stored it in a list, but I don’t know how to replace the column names.

Answer accepted (score 1585)

Just assign it to the .columns attribute:

>>> df = pd.DataFrame({'$a':[1,2], '$b': [10,20]})
>>> df.columns = ['a', 'b']
>>> df
   a   b
0  1  10
1  2  20

Answer 2 (score 2506)

RENAME SPECIFIC COLUMNS

Use the df.rename() function and refer the columns to be renamed. Not all the columns have to be renamed:

df = df.rename(columns={'oldName1': 'newName1', 'oldName2': 'newName2'})
# Or rename the existing DataFrame (rather than creating a copy) 
df.rename(columns={'oldName1': 'newName1', 'oldName2': 'newName2'}, inplace=True)

Minimal Code Example

df = pd.DataFrame('x', index=range(3), columns=list('abcde'))
df

   a  b  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x

The following methods all work and produce the same output:

df2 = df.rename({'a': 'X', 'b': 'Y'}, axis=1)  # new method
df2 = df.rename({'a': 'X', 'b': 'Y'}, axis='columns')
df2 = df.rename(columns={'a': 'X', 'b': 'Y'})  # old method  

df2

   X  Y  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x

Remember to assign the result back, as the modification is not-inplace. Alternatively, specify inplace=True:

df.rename({'a': 'X', 'b': 'Y'}, axis=1, inplace=True)
df

   X  Y  c  d  e
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x

From v0.25, you can also specify errors='raise' to raise errors if an invalid column-to-rename is specified. See v0.25 rename() docs.

REASSIGN COLUMN HEADERS

Use df.set_axis() with axis=1 and inplace=False (to return a copy).

df2 = df.set_axis(['V', 'W', 'X', 'Y', 'Z'], axis=1, inplace=False)
df2

   V  W  X  Y  Z
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x

This returns a copy, but you can modify the DataFrame in-place by setting inplace=True (this is the default behaviour for versions <=0.24 but is likely to change in the future).

You can also assign headers directly:

df.columns = ['V', 'W', 'X', 'Y', 'Z']
df

   V  W  X  Y  Z
0  x  x  x  x  x
1  x  x  x  x  x
2  x  x  x  x  x

Answer 3 (score 364)

The rename method can take a function, for example:

In [11]: df.columns
Out[11]: Index([u'$a', u'$b', u'$c', u'$d', u'$e'], dtype=object)

In [12]: df.rename(columns=lambda x: x[1:], inplace=True)

In [13]: df.columns
Out[13]: Index([u'a', u'b', u'c', u'd', u'e'], dtype=object)

Question

How do I see the type of a variable whether it is unsigned 32 bit, signed 16 bit, etc.?

How do I view it?

Answer 2 (score 1224)

Python doesn’t have the same types as C/C++, which appears to be your question.

Try this:

>>> i = 123
>>> type(i)
<type 'int'>
>>> type(i) is int
True
>>> i = 123456789L
>>> type(i)
<type 'long'>
>>> type(i) is long
True
>>> i = 123.456
>>> type(i)
<type 'float'>
>>> type(i) is float
True

The distinction between int and long goes away in Python 3.0, though.

Answer 3 (score 403)

You may be looking for the type() function.

See the examples below, but there’s no “unsigned” type in Python just like Java.

Positive integer:

>>> v = 10
>>> type(v)
<type 'int'>

Large positive integer:

>>> v = 100000000000000
>>> type(v)
<type 'long'>

Negative integer:

>>> v = -10
>>> type(v)
<type 'int'>

Literal sequence of characters:

>>> v = 'hi'
>>> type(v)
<type 'str'>

Floating point integer:

>>> v = 3.14159
>>> type(v)
<type 'float'>

Question

Is there a way to convert a string from uppercase, or even part uppercase to lowercase?

For example, “Kilometers” → “kilometers”.

Answer accepted (score 2880)

Use .lower() - For example:

s = "Kilometer"
print(s.lower())

The official 2.x documentation is here: str.lower()
The official 3.x documentation is here: str.lower()

Answer 2 (score 197)

How to convert string to lowercase in Python?

Is there any way to convert an entire user inputted string from uppercase, or even part uppercase to lowercase?

E.g. Kilometers –> kilometers

The canonical Pythonic way of doing this is

>>> 'Kilometers'.lower()
'kilometers'

However, if the purpose is to do case insensitive matching, you should use case-folding:

>>> 'Kilometers'.casefold()
'kilometers'

Here’s why:

>>> "Maße".casefold()
'masse'
>>> "Maße".lower()
'maße'
>>> "MASSE" == "Maße"
False
>>> "MASSE".lower() == "Maße".lower()
False
>>> "MASSE".casefold() == "Maße".casefold()
True

This is a str method in Python 3, but in Python 2, you’ll want to look at the PyICU or py2casefold - several answers address this here.

Unicode Python 3

Python 3 handles plain string literals as unicode:

>>> string = 'Километр'
>>> string
'Километр'
>>> string.lower()
'километр'

Python 2, plain string literals are bytes

In Python 2, the below, pasted into a shell, encodes the literal as a string of bytes, using utf-8.

And lower doesn’t map any changes that bytes would be aware of, so we get the same string.

>>> string = 'Километр'
>>> string
'\xd0\x9a\xd0\xb8\xd0\xbb\xd0\xbe\xd0\xbc\xd0\xb5\xd1\x82\xd1\x80'
>>> string.lower()
'\xd0\x9a\xd0\xb8\xd0\xbb\xd0\xbe\xd0\xbc\xd0\xb5\xd1\x82\xd1\x80'
>>> print string.lower()
Километр

In scripts, Python will object to non-ascii (as of Python 2.5, and warning in Python 2.4) bytes being in a string with no encoding given, since the intended coding would be ambiguous. For more on that, see the Unicode how-to in the docs and PEP 263

Use Unicode literals, not str literals

So we need a unicode string to handle this conversion, accomplished easily with a unicode string literal, which disambiguates with a u prefix (and note the u prefix also works in Python 3):

>>> unicode_literal = u'Километр'
>>> print(unicode_literal.lower())
километр

Note that the bytes are completely different from the str bytes - the escape character is '\u' followed by the 2-byte width, or 16 bit representation of these unicode letters:

>>> unicode_literal
u'\u041a\u0438\u043b\u043e\u043c\u0435\u0442\u0440'
>>> unicode_literal.lower()
u'\u043a\u0438\u043b\u043e\u043c\u0435\u0442\u0440'

Now if we only have it in the form of a str, we need to convert it to unicode. Python’s Unicode type is a universal encoding format that has many advantages relative to most other encodings. We can either use the unicode constructor or str.decode method with the codec to convert the str to unicode:

>>> unicode_from_string = unicode(string, 'utf-8') # "encoding" unicode from string
>>> print(unicode_from_string.lower())
километр
>>> string_to_unicode = string.decode('utf-8') 
>>> print(string_to_unicode.lower())
километр
>>> unicode_from_string == string_to_unicode == unicode_literal
True

Both methods convert to the unicode type - and same as the unicode_literal.

Best Practice, use Unicode

It is recommended that you always work with text in Unicode.

Software should only work with Unicode strings internally, converting to a particular encoding on output.

Can encode back when necessary

However, to get the lowercase back in type str, encode the python string to utf-8 again:

>>> print string
Километр
>>> string
'\xd0\x9a\xd0\xb8\xd0\xbb\xd0\xbe\xd0\xbc\xd0\xb5\xd1\x82\xd1\x80'
>>> string.decode('utf-8')
u'\u041a\u0438\u043b\u043e\u043c\u0435\u0442\u0440'
>>> string.decode('utf-8').lower()
u'\u043a\u0438\u043b\u043e\u043c\u0435\u0442\u0440'
>>> string.decode('utf-8').lower().encode('utf-8')
'\xd0\xba\xd0\xb8\xd0\xbb\xd0\xbe\xd0\xbc\xd0\xb5\xd1\x82\xd1\x80'
>>> print string.decode('utf-8').lower().encode('utf-8')
километр

So in Python 2, Unicode can encode into Python strings, and Python strings can decode into the Unicode type.

Answer 3 (score 195)

With Python 2, this doesn’t work for non-English words in UTF-8. In this case decode('utf-8') can help:

>>> s='Километр'
>>> print s.lower()
Километр
>>> print s.decode('utf-8').lower()
километр

Question

How can I convert a list to a string using Python?

Answer 2 (score 1203)

By using ''.join

list1 = ['1', '2', '3']
str1 = ''.join(list1)

Or if the list is of integers, convert the elements before joining them.

list1 = [1, 2, 3]
str1 = ''.join(str(e) for e in list1)

Answer 3 (score 255)

>>> L = [1,2,3]       
>>> " ".join(str(x) for x in L)
'1 2 3'

Question

How to select rows from a DataFrame based on values in some column in pandas?

In SQL, I would use:

SELECT *
FROM table
WHERE colume_name = some_value

I tried to look at pandas documentation but did not immediately find the answer.

Answer accepted (score 3029)

To select rows whose column value equals a scalar, some_value, use ==:

df.loc[df['column_name'] == some_value]

To select rows whose column value is in an iterable, some_values, use isin:

df.loc[df['column_name'].isin(some_values)]

Combine multiple conditions with &:

df.loc[(df['column_name'] >= A) & (df['column_name'] <= B)]

Note the parentheses. Due to Python’s operator precedence rules, & binds more tightly than <= and >=. Thus, the parentheses in the last example are necessary. Without the parentheses

df['column_name'] >= A & df['column_name'] <= B

is parsed as

df['column_name'] >= (A & df['column_name']) <= B

which results in a Truth value of a Series is ambiguous error.

To select rows whose column value does not equal some_value, use !=:

df.loc[df['column_name'] != some_value]

isin returns a boolean Series, so to select rows whose value is not in some_values, negate the boolean Series using ~:

df.loc[~df['column_name'].isin(some_values)]

For example,

import pandas as pd
import numpy as np
df = pd.DataFrame({'A': 'foo bar foo bar foo bar foo foo'.split(),
                   'B': 'one one two three two two one three'.split(),
                   'C': np.arange(8), 'D': np.arange(8) * 2})
print(df)
#      A      B  C   D
# 0  foo    one  0   0
# 1  bar    one  1   2
# 2  foo    two  2   4
# 3  bar  three  3   6
# 4  foo    two  4   8
# 5  bar    two  5  10
# 6  foo    one  6  12
# 7  foo  three  7  14

print(df.loc[df['A'] == 'foo'])

yields

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

If you have multiple values you want to include, put them in a list (or more generally, any iterable) and use isin:

print(df.loc[df['B'].isin(['one','three'])])

yields

     A      B  C   D
0  foo    one  0   0
1  bar    one  1   2
3  bar  three  3   6
6  foo    one  6  12
7  foo  three  7  14

Note, however, that if you wish to do this many times, it is more efficient to make an index first, and then use df.loc:

df = df.set_index(['B'])
print(df.loc['one'])

yields

       A  C   D
B              
one  foo  0   0
one  bar  1   2
one  foo  6  12

or, to include multiple values from the index use df.index.isin:

df.loc[df.index.isin(['one','two'])]

yields

       A  C   D
B              
one  foo  0   0
one  bar  1   2
two  foo  2   4
two  foo  4   8
two  bar  5  10
one  foo  6  12

Answer 2 (score 244)

tl;dr

The pandas equivalent to

select * from table where column_name = some_value

is

table[table.column_name == some_value]

Multiple conditions:

table[(table.column_name == some_value) | (table.column_name2 == some_value2)]

or

table.query('column_name == some_value | column_name2 == some_value2')

Code example

import pandas as pd

# Create data set
d = {'foo':[100, 111, 222], 
     'bar':[333, 444, 555]}
df = pd.DataFrame(d)

# Full dataframe:
df

# Shows:
#    bar   foo 
# 0  333   100
# 1  444   111
# 2  555   222

# Output only the row(s) in df where foo is 222:
df[df.foo == 222]

# Shows:
#    bar  foo
# 2  555  222

In the above code it is the line df[df.foo == 222] that gives the rows based on the column value, 222 in this case.

Multiple conditions are also possible:

df[(df.foo == 222) | (df.bar == 444)]
#    bar  foo
# 1  444  111
# 2  555  222

But at that point I would recommend using the query function, since it’s less verbose and yields the same result:

df.query('foo == 222 | bar == 444')

Answer 3 (score 205)

There are several ways to select rows from a pandas data frame:

1. Boolean indexing (df[df['col'] == value] )
2. Positional indexing (df.iloc[...])
3. Label indexing (df.xs(...))
4. df.query(...) API

Below I show you examples of each, with advice when to use certain techniques. Assume our criterion is column 'A' == 'foo'

(Note on performance: For each base type, we can keep things simple by using the pandas API or we can venture outside the API, usually into numpy, and speed things up.)

Setup
The first thing we’ll need is to identify a condition that will act as our criterion for selecting rows. We’ll start with the OP’s case column_name == some_value, and include some other common use cases.

Borrowing from @unutbu:

import pandas as pd, numpy as np

df = pd.DataFrame({'A': 'foo bar foo bar foo bar foo foo'.split(),
                   'B': 'one one two three two two one three'.split(),
                   'C': np.arange(8), 'D': np.arange(8) * 2})

1. Boolean indexing

… Boolean indexing requires finding the true value of each row’s 'A' column being equal to 'foo', then using those truth values to identify which rows to keep. Typically, we’d name this series, an array of truth values, mask. We’ll do so here as well.

mask = df['A'] == 'foo'

We can then use this mask to slice or index the data frame

df[mask]

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

This is one of the simplest ways to accomplish this task and if performance or intuitiveness isn’t an issue, this should be your chosen method. However, if performance is a concern, then you might want to consider an alternative way of creating the mask.

2. Positional indexing

Positional indexing (df.iloc[...]) has its use cases, but this isn’t one of them. In order to identify where to slice, we first need to perform the same boolean analysis we did above. This leaves us performing one extra step to accomplish the same task.

mask = df['A'] == 'foo'
pos = np.flatnonzero(mask)
df.iloc[pos]

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

3. Label indexing

Label indexing can be very handy, but in this case, we are again doing more work for no benefit

df.set_index('A', append=True, drop=False).xs('foo', level=1)

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

4. df.query() API

pd.DataFrame.query is a very elegant/intuitive way to perform this task, but is often slower. However, if you pay attention to the timings below, for large data, the query is very efficient. More so than the standard approach and of similar magnitude as my best suggestion.

df.query('A == "foo"')

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

My preference is to use the Boolean mask

Actual improvements can be made by modifying how we create our Boolean mask.

mask alternative 1
Use the underlying numpy array and forgo the overhead of creating another pd.Series

mask = df['A'].values == 'foo'

I’ll show more complete time tests at the end, but just take a look at the performance gains we get using the sample data frame. First, we look at the difference in creating the mask

%timeit mask = df['A'].values == 'foo'
%timeit mask = df['A'] == 'foo'

5.84 µs ± 195 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
166 µs ± 4.45 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Evaluating the mask with the numpy array is ~ 30 times faster. This is partly due to numpy evaluation often being faster. It is also partly due to the lack of overhead necessary to build an index and a corresponding pd.Series object.

Next, we’ll look at the timing for slicing with one mask versus the other.

mask = df['A'].values == 'foo'
%timeit df[mask]
mask = df['A'] == 'foo'
%timeit df[mask]

219 µs ± 12.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
239 µs ± 7.03 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The performance gains aren’t as pronounced. We’ll see if this holds up over more robust testing.

mask alternative 2
We could have reconstructed the data frame as well. There is a big caveat when reconstructing a dataframe—you must take care of the dtypes when doing so!

Instead of df[mask] we will do this

pd.DataFrame(df.values[mask], df.index[mask], df.columns).astype(df.dtypes)

If the data frame is of mixed type, which our example is, then when we get df.values the resulting array is of dtype object and consequently, all columns of the new data frame will be of dtype object. Thus requiring the astype(df.dtypes) and killing any potential performance gains.

%timeit df[m]
%timeit pd.DataFrame(df.values[mask], df.index[mask], df.columns).astype(df.dtypes)

216 µs ± 10.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.43 ms ± 39.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

However, if the data frame is not of mixed type, this is a very useful way to do it.

Given

np.random.seed([3,1415])
d1 = pd.DataFrame(np.random.randint(10, size=(10, 5)), columns=list('ABCDE'))

d1

   A  B  C  D  E
0  0  2  7  3  8
1  7  0  6  8  6
2  0  2  0  4  9
3  7  3  2  4  3
4  3  6  7  7  4
5  5  3  7  5  9
6  8  7  6  4  7
7  6  2  6  6  5
8  2  8  7  5  8
9  4  7  6  1  5    

%%timeit
mask = d1['A'].values == 7
d1[mask]

179 µs ± 8.73 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Versus

%%timeit
mask = d1['A'].values == 7
pd.DataFrame(d1.values[mask], d1.index[mask], d1.columns)

87 µs ± 5.12 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

We cut the time in half.

mask alternative 3
@unutbu also shows us how to use pd.Series.isin to account for each element of df['A'] being in a set of values. This evaluates to the same thing if our set of values is a set of one value, namely 'foo'. But it also generalizes to include larger sets of values if needed. Turns out, this is still pretty fast even though it is a more general solution. The only real loss is in intuitiveness for those not familiar with the concept.

mask = df['A'].isin(['foo'])
df[mask]

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

However, as before, we can utilize numpy to improve performance while sacrificing virtually nothing. We’ll use np.in1d

mask = np.in1d(df['A'].values, ['foo'])
df[mask]

     A      B  C   D
0  foo    one  0   0
2  foo    two  2   4
4  foo    two  4   8
6  foo    one  6  12
7  foo  three  7  14

Timing
I’ll include other concepts mentioned in other posts as well for reference.
Code Below

Each Column in this table represents a different length data frame over which we test each function. Each column shows relative time taken, with the fastest function given a base index of 1.0.

res.div(res.min())

                         10        30        100       300       1000      3000      10000     30000
mask_standard         2.156872  1.850663  2.034149  2.166312  2.164541  3.090372  2.981326  3.131151
mask_standard_loc     1.879035  1.782366  1.988823  2.338112  2.361391  3.036131  2.998112  2.990103
mask_with_values      1.010166  1.000000  1.005113  1.026363  1.028698  1.293741  1.007824  1.016919
mask_with_values_loc  1.196843  1.300228  1.000000  1.000000  1.038989  1.219233  1.037020  1.000000
query                 4.997304  4.765554  5.934096  4.500559  2.997924  2.397013  1.680447  1.398190
xs_label              4.124597  4.272363  5.596152  4.295331  4.676591  5.710680  6.032809  8.950255
mask_with_isin        1.674055  1.679935  1.847972  1.724183  1.345111  1.405231  1.253554  1.264760
mask_with_in1d        1.000000  1.083807  1.220493  1.101929  1.000000  1.000000  1.000000  1.144175

You’ll notice that fastest times seem to be shared between mask_with_values and mask_with_in1d

res.T.plot(loglog=True)