Hướng dẫn python json dump one per line - python json dump một trên mỗi dòng

For fun I adapted my answer to another somewhat related question to make it do what you want. Note that currently it only changes the formatting of a

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
7 if it's in a list.

import _ctypes
import json
import re

class OneDictPerLine(object):
    def __init__(self, value):
        self.value = value
    def __repr__(self):
        if not isinstance(self.value, list):
            return repr(self.value)
        else:  # Sort the representation of any dicts in the list.
            reps = ('{{{}}}'.format(', '.join(
                        ('{!r}: {}'.format(k, v) for k, v in sorted(v.items()))
                    )) if isinstance(v, dict)
                        else
                    repr(v) for v in self.value)
            return '[' + ',\n'.join(reps) + ']'


def di(obj_id):
    """ Reverse of id() function. """
    # from https://stackoverflow.com/a/15012814/355230
    return _ctypes.PyObj_FromPtr(obj_id)


class MyEncoder(json.JSONEncoder):
    FORMAT_SPEC = "@@{}@@"
    regex = re.compile(FORMAT_SPEC.format(r"(\d+)"))

    def default(self, obj):
        return (self.FORMAT_SPEC.format(id(obj)) if isinstance(obj, OneDictPerLine)
                else super(MyEncoder, self).default(obj))

    def encode(self, obj):
        format_spec = self.FORMAT_SPEC  # Local var to expedite access.
        json_repr = super(MyEncoder, self).encode(obj)  # Default JSON repr.

        # Replace any marked-up object ids in the JSON repr with the value
        # returned from the repr() of the corresponding Python object.
        for match in self.regex.finditer(json_repr):
            id = int(match.group(1))
            # Replace marked-up id with actual Python object repr().
            json_repr = json_repr.replace(
                       '"{}"'.format(format_spec.format(id)), repr(di(id)))

        return json_repr

Sample usage:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))

Output:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]

json.dump(s) & json.load(s)

Hướng dẫn python json dump one per line - python json dump một trên mỗi dòng

Hướng dẫn python json dump one per line - python json dump một trên mỗi dòng






bogotobogo.com site search:


json.load(s) & json.dump(s)

Hướng dẫn python json dump one per line - python json dump một trên mỗi dòng

There are two ways of reading in (load/loads) the following json file, in.json:

{"alpha": 1, "beta": 2}    

  1. string:
    import json
    
    io = open("in.json","r")
    string = io.read()
    # json.loads(str)
    dictionary = json.loads(string)
    
    # or one-liner
    # dictionary = json.loads(open("in.json","r").read())
    
    print(dictionary)
    

  2. dictionary:
    import json
    # json.load(_io)
    io = open("in.json","r")
    dictionary = json.load(io)
    
    # or one-liner
    # dictionary = json.load(open("in.json","r"))
    
    print(dictionary)
    

Both will print out:

{'alpha': 1, 'beta': 2}

Note that while the json.loads() requires string, load(s,...), the json.load() requires file descriptor, load(fp...).


Similarly, we can write a (dump/dumps) json file:

  1. string:
    import json
    d = {'alpha': 1, 'beta': 2}
    s = json.dumps(d)
    open("out.json","w").write(s)
    

  2. dictionary:
    import json
    d = {'alpha': 1, 'beta': 2}
    json.dump(d, open("out.json","w"))
    

Note that the json.dump() requires file descriptor as well as an obj, dump(obj, fp...).




In the following example, we'll convert Python dictionary to JSON and write it to a text file. Then, we'll read in back from the file and play with it.

Initially we'll construct Python dictionary like this:

# Four Fundamental Forces with JSON
d = {}

d ["gravity"] = {
"mediator":"gravitons",
"relative strength" : "1",
"range" : "infinity"
}
d ["weak"] = {
"mediator":"W/Z bosons",
"relative strength" : "10^25",
"range" : "10^-18"
}
d ["electromagnetic"] = {
"mediator":"photons",
"relative strength" : "10^36",
"range" : "infinity"
}
d ["strong"] = {
"mediator":"gluons",
"relative strength" : "10^38",
"range" : "10^-15"
}

print(d)

The output looks like this:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
0

Now, we want to convert the dictionary to a string using json.dumps:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
1

Output:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
2

Note that the "json.dumps()" returns a string as indicated by the "s" at the end of "dumps". This process is called encoding.

Let's write it to a file:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
3

Now that the file is written. Let's reads it back and decoding the JSON-encoded string back into a Python dictionary data structure:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
4

Let's play with the dictionary a little bit.

What's the relative strength of electromagnetic compared to gravity?

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
5

Who's the mediator for "strong" force?

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
6

Ok, here is the full code:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
7

If we prefer working with files instead of strings, we may want to use json.dump() / json.load() to encode / decode JSON data using the data from the previous example:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
8

Here is another example (json.dump()/json.load()) using simpler data:

# Sample usage
data = [
    {"key01":"value","key02":"value"},
    {"key11":"value","key12":"value"},
    {"key21":"value","key22":"value"},
    {'key{:02d}:"value"'.format(k) for k in range(100)}
]

print(json.dumps(OneDictPerLine(data), cls=MyEncoder))
9

Usage for string version: json.loads()/json.dumps():

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
0

The following example sends a syslog to logstash fargate containers behind AWS NLB:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
1

note that for the HOST, we can also use FQDN instead of the NLB's domain name.

Also, as usual, instead of the long line of code, we may want to use a simple linux command,

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
8:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
2

If the NLB listener protocol is TLS, we can use

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
9 echo to the TLS NLB:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
3

Another example: AWS API response.

When we make an AWS API call, the response can be an invalid json due to datetime:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
4

We need to serialize it (ow to overcome “datetime.datetime not JSON serializable”?).

Here is a boto3 code for an API call to EC2 describe:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
5

The r.json with jq looks like this:

[{'key01': value, 'key02': value},
{'key11': value, 'key12': value},
{'key21': value, 'key22': value},
{'key93:"value"', 'key05:"value"', 'key00:"value"', 'key33:"value"', 'key55:"value"', 'key91:"value"', 'key18:"value"', 'key76:"value"', 'key25:"value"', 'key72:"value"', 'key21:"value"', 'key54:"value"', 'key12:"value"', 'key61:"value"', 'key96:"value"', 'key87:"value"', 'key71:"value"', 'key03:"value"', 'key66:"value"', 'key58:"value"', 'key85:"value"', 'key11:"value"', 'key64:"value"', 'key75:"value"', 'key27:"value"', 'key86:"value"', 'key29:"value"', 'key31:"value"', 'key69:"value"', 'key15:"value"', 'key62:"value"', 'key45:"value"', 'key49:"value"', 'key40:"value"', 'key39:"value"', 'key78:"value"', 'key98:"value"', 'key28:"value"', 'key19:"value"', 'key42:"value"', 'key60:"value"', 'key04:"value"', 'key84:"value"', 'key56:"value"', 'key83:"value"', 'key10:"value"', 'key34:"value"', 'key77:"value"', 'key80:"value"', 'key68:"value"', 'key99:"value"', 'key38:"value"', 'key67:"value"', 'key59:"value"', 'key52:"value"', 'key57:"value"', 'key23:"value"', 'key14:"value"', 'key26:"value"', 'key90:"value"', 'key09:"value"', 'key07:"value"', 'key35:"value"', 'key73:"value"', 'key41:"value"', 'key17:"value"', 'key48:"value"', 'key44:"value"', 'key82:"value"', 'key65:"value"', 'key47:"value"', 'key95:"value"', 'key88:"value"', 'key97:"value"', 'key63:"value"', 'key22:"value"', 'key51:"value"', 'key50:"value"', 'key36:"value"', 'key06:"value"', 'key30:"value"', 'key32:"value"', 'key08:"value"', 'key79:"value"', 'key89:"value"', 'key20:"value"', 'key70:"value"', 'key46:"value"', 'key94:"value"', 'key53:"value"', 'key92:"value"', 'key81:"value"', 'key13:"value"', 'key43:"value"', 'key24:"value"', 'key16:"value"', 'key02:"value"', 'key74:"value"', 'key01:"value"', 'key37:"value"'}]
6



Python tutorial


Python Home

Introduction

Running Python Programs (os, sys, import)

Modules and IDLE (Import, Reload, exec)

Object Types - Numbers, Strings, and None

Strings - Escape Sequence, Raw String, and Slicing

Strings - Methods

Formatting Strings - expressions and method calls

Files and os.path

Traversing directories recursively

Subprocess Module

Regular Expressions with Python

Regular Expressions Cheat Sheet

Object Types - Lists

Object Types - Dictionaries and Tuples

Functions def, *args, **kargs

Functions lambda

Built-in Functions

map, filter, and reduce

Decorators

List Comprehension

Sets (union/intersection) and itertools - Jaccard coefficient and shingling to check plagiarism

Hashing (Hash tables and hashlib)

Dictionary Comprehension with zip

The yield keyword

Generator Functions and Expressions

generator.send() method

Iterators

Classes and Instances (__init__, __call__, etc.)

if__name__ == '__main__'

argparse

Exceptions

@static method vs class method

Private attributes and private methods

bits, bytes, bitstring, and constBitStream

json.dump(s) and json.load(s)

Python Object Serialization - pickle and json

Python Object Serialization - yaml and json

Priority queue and heap queue data structure

Graph data structure

Dijkstra's shortest path algorithm

Prim's spanning tree algorithm

Closure

Functional programming in Python

Remote running a local file using ssh

SQLite 3 - A. Connecting to DB, create/drop table, and insert data into a table

SQLite 3 - B. Selecting, updating and deleting data

MongoDB with PyMongo I - Installing MongoDB ...

Python HTTP Web Services - urllib, httplib2

Web scraping with Selenium for checking domain availability

REST API : Http Requests for Humans with Flask

Blog app with Tornado

Multithreading ...

Python Network Programming I - Basic Server / Client : A Basics

Python Network Programming I - Basic Server / Client : B File Transfer

Python Network Programming II - Chat Server / Client

Python Network Programming III - Echo Server using socketserver network framework

Python Network Programming IV - Asynchronous Request Handling : ThreadingMixIn and ForkingMixIn

Python Coding Questions I

Python Coding Questions II

Python Coding Questions III

Python Coding Questions IV

Python Coding Questions V

Python Coding Questions VI

Python Coding Questions VII

Python Coding Questions VIII

Image processing with Python image library Pillow

Python and C++ with SIP

PyDev with Eclipse

Matplotlib

Redis with Python

NumPy array basics A

NumPy Matrix and Linear Algebra

Pandas with NumPy and Matplotlib

Celluar Automata

Batch gradient descent algorithm

Longest Common Substring Algorithm

Python Unit Test - TDD using unittest.TestCase class

Simple tool - Google page ranking by keywords

Google App Hello World

Google App webapp2 and WSGI

Uploading Google App Hello World

Python 2 vs Python 3

virtualenv and virtualenvwrapper

Uploading a big file to AWS S3 using boto module

Scheduled stopping and starting an AWS instance

Cloudera CDH5 - Scheduled stopping and starting services

Removing Cloud Files - Rackspace API with curl and subprocess

Checking if a process is running/hanging and stop/run a scheduled task on Windows

Apache Spark 1.3 with PySpark (Spark Python API) Shell

Apache Spark 1.2 Streaming

bottle 0.12.7 - Fast and simple WSGI-micro framework for small web-applications ...

Flask app with Apache WSGI on Ubuntu14/CentOS7 ...

Fabric - streamlining the use of SSH for application deployment

Ansible Quick Preview - Setting up web servers with Nginx, configure enviroments, and deploy an App

Neural Networks with backpropagation for XOR using one hidden layer

NLP - NLTK (Natural Language Toolkit) ...

RabbitMQ(Message broker server) and Celery(Task queue) ...

OpenCV3 and Matplotlib ...

Simple tool - Concatenating slides using FFmpeg ...

iPython - Signal Processing with NumPy

iPython and Jupyter - Install Jupyter, iPython Notebook, drawing with Matplotlib, and publishing it to Github

iPython and Jupyter Notebook with Embedded D3.js

Downloading YouTube videos using youtube-dl embedded with Python

Machine Learning : scikit-learn ...

Django 1.6/1.8 Web Framework ...