Determine the top 10 words in a text (POS1: 2A, 2C)
Task 1 Write a function
hauf (string) which (number of words) the 10 most common words in the string string and determines a list of tuples of the kind
returns (the length of the list is <= 10). Die Liste muss so sortiert sein, dass das Häufigste Wort ganz am Anfang ist. Haben Worte die selben Häufigkeiten, so haben diese alphabetisch sortiert zu sein.
Task 2 Write a function
histogram (list, length = 60) , one consisting of the passed list Histogram is generated and delivered as a string. A "line" should look like this:
the 123 ************
Where the word for 14 characters, the number three characters must be provided. Behind a number of asterisks equal to the number based on the maximum (highest number in the list) and the width is equivalent to (
round (no. / * max width) ).
Task 3 Write a function
main () , all of which will open on the command line specified files and determines the overall frequencies (ie
hauf () with the entire text read calls) and determined with the The histogram data outputs.
If not specify any files, it should be read from stdin (up
EOF).
sample call with the test data from the / default directory:
hp @ L111 ~ / workspace / python $ python3 haufdiagramm.py testdaten.txt
the 89 ****************** ******************************************
the 61 ***** ************************************
and 43 *********** ******************
histogram 40 ***************************
in 28 *******************
for 17 *********** ***********
is 17
the 16 ***********
the 15 ********** *********
a 14
Possible solution
I am the individual functions explain:
import sys
This import is needed for the command-line arguments.
following function determines the word-frequencies, the results sorted by frequency and return the first 10 elements.
def hauf (string):
"" "identifies the 10 most common words in 'string' and returns the list of
tuple (word, count)" ""
dict = {} for word in string.split
():
split () without parameter splits a string into a list of "words" with a "word" is a sequence of characters with no spaces, tab and newline. In other words, spaces, tabs and newlines are separators.
word = wort.rstrip ('.,;?!")') # remove punctuation
word = wort.lstrip ('"(') # single quote and bracket may be forward
lstrip () and
rstrip () remove all characters in brackets
l left or r ight . In our case, all punctuation marks, which may directly in front of a word or a word.
The two lines, you could probably also be combined into.
word = wort.rstrip ('.,;?!")'). lstrip ('"(') The first line checks whether the string actually is a word (letters only)
if wort.isalpha ().
if word in dict:
dict [word] + = 1 else
:
dict [word] = 1
above 4, last line first check whether the word already included in the Dictionary. If so, then the stored number is increased, if not, then this is a word a
saved.
l = list (dict.items ())
Creates a list of tuples
(word, number) from the dictionary.
l.sort (key = lambda x: (-x [1], x [0]))
sort () filter a list by the natural order. The parameter key
a function can be passed to sort based on their return value. The function is called with each list item. So you could write a function, each of the second element of the tuple (The list is in this case, so word of tuples of type
("number) ) returns. There will be sorted in descending order, either the value -1
are multiplied (
- [x1] ) or you must sort the sort
call (reverse = True) . The name of this function, you pass it by
key = sortfunktion .
This function might be:
def sorting function (x): return (-x [1], x [0]) lambda x: defines an anonymous function (function without a name) with the parameter
x , the same will be defined and passed on the spot can.
l = l [00:10] l
return from the last two lines intersect the first 10 elements of the list and give it back.
This could be done only with a statement:
return l [: 10] .
def stars (num, max, width):
"" "returns string with as many stars as num / * max width" ""
return "*" * round (num / max * width)
given You do not need to say too much.
The following function produces the required in Task 2
histogram. To find
an appropriate scale for the required length = 60
wide, the maximum of the values is found. This must of course be the second element of the first tuple of the list, if any are tuples in the list.
def histogram (list, length = 60):
"" "generates histogram and returns it" ""
if len (list)> 0:
max = list [0] [1]
ret = ""
for word, number in list:
This loop goes through each tuple of the list, with the same tuple in
word and
number "unpacked" is.
ret + = "%-14s% 3d% s \\ n"% (word, number, stars (number, max, width))
Here is each line of the string ret
appended String formatting with
is applied.
return ret
The following two functions is used only for testing with fixed strings and returns a string with specified number of words.
def testStr ():
"" "provides test input" ""
s = "frequency" * 50 + "for the" * 123 + "the" * 100 + "house" * 40
s + = "mouse" * 38 + "Case" * 34 + "kg" * 22 + "point" * 18
s + = "Right" * 16 + "Ten" * 10
s + = "nothing" * 5 + "23" * 50 + "fairly" * 3
s return
def test ():
"" "Test ..."""
list = hauf (testStr ())
print (histogram (list, 12.3))
This function checks whether any command line arguments are available. Are there any, then all arguments as files are opened for reading and
read () read in full to the string input
attached.
there is no file name on the command line, is read by
sys.stdin.read input = () from standard input.
The last line calls on the two main functions, and outputs the result. It lacks
still checking if a file can be opened or even exists. This part may supplement the reader himself.
def main ():
"" "open file or read from stdin" ""
if len (sys.argv)> 1:
input = "" for arg in sys.argv
[1]:
f = open (arg)
input + = "" + f.read () f.close
()
else
sys.stdin.read input = ()
print (histogram (hauf (input)))
The following lines are used to calling the function
main () (or
test () for testing), if this script was invoked as a program.
if __name__ == '__main__':
# test ()
main ()
Full program (without headers)
import sys def
hauf (string):
"" "identifies the 10 most common words in 'string' and list of supplies
tuple (word, count) "" "
dict = {} for word in string.split
(): word =
wort.rstrip ('.,;?!")') # remove punctuation
word = wort.lstrip ('"(') # single quote and bracket may be ahead if
wort.isalpha ():
word if dict:
dict [word] + = 1 else
:
dict [word] = 1
l = list (dict.items ()) l.sort
(key = lambda x: (-x [1], x [0]))
l = l [12:10]
return l def
stars (num, max, width):
"" "returns string with as many stars as num / * max width" ""
return "*" * round (num / max * width)
def histogram (list, length = 60):
"" "generates histogram and returns it" ""
if len (list)> 0:
max = list [0] [1]
ret = "" for
word, number in list:
ret + = "%-14s% 3d% s \\ n"% (word, number, Reference (number, max, width)) return
ret
def testStr ():
"" "provides test input" ""
s = "frequency" * 50 + "for the" * 123 + "the" * 100 + "House" * 40
s + = "mouse" * 38 + "case" * 34 + "kg" * 22 + "point" * 18
s + = "Right" * 16 + "Ten" * 10 + s
= "nothing" * 5 + "23" * 50 + "fairly" * 3
s return
def test ():
"" "Test ..."""
list = hauf (testStr ()) print
(histogram (list, 12.3))
def main ():
"opens" " Reads file or stdin "" "
if len (sys.argv)> 1:
input =" "for arg
in sys.argv [1:]: open
f = (arg)
input + =" "+ f.read () f.close
()
else
sys.stdin.read input = ()
print (histogram (hauf (input)))
if __name__ == '__main__':
# test ()
main ()