All Questions
Tagged with python-3.x pandas
27,482
questions
-1votes
0answers
10views
Ragged nested array need to be converted to array of float
I have a ragged nested array with class 'numpy.ndarray' which I want to convert it to a string of float which I already try to use np.asfarray to change it. the problem is that, my Jupyter Notebook ...
0votes
0answers
8views
Dash Plotly error TypeError: Object of type DataFrame is not JSON serializable
Hello I am working with Dash for making dashboard.
Below is my code.
I tried to fix the error but not able to fix, Can anyone look into this?
on chrome i am getting. Error loading layout
I am getting ...
- 23
0votes
0answers
10views
Ragged Nested Numpy Array to Tensor
I have ragged nested arrays of .npy file that I have processed. The error is that it my model failed to convert the NumPy array to a Tensor with the remark 'Unsupported object type list') I have try ...
0votes
0answers
7views
How to crate a Hybrid Work Model roster simulation
My office is currently manually creating Hybrid work model Rosters. Can we automate the process in Python/Pandas?
Rules for creating a roster is:
for 2 weeks (10 working days) people should be in ...
0votes
1answer
41views
Pandas get rows with subtracting rows as a index
I have a function in python that gets the two rows for a yfinance dataframe of tickers. I am doing a calculation, where I make a row that holds the value obtained from subtracting the value, then ...
- 415
1vote
2answers
15views
the ways of using zip function to generate the cell location based on the row and column information
I created the following dataframe, and would like to identify the cells which are Null,
import pandas as pd
import numpy as np
data = [{'a': 1, 'b': 2, 'c':3},
{'a':10, 'b': np.NaN, 'c':"&...
- 4,679
0votes
1answer
18views
How to change float to date type in python? (ValueError: day is out of range for month)
I have the following column:
0 3012022.0
1 3012022.0
2 3012022.0
3 3012022.0
4 3012022.0
...
351 24032022.0
352 24032022.0
df.Data = df.Data.astype('...
- 57
1vote
3answers
37views
Formatting Multiple Columns by Name using .loc
My goal is to state a list of columns by name that I want to apply the formatting to.
The section that is commented out is ideally what I am after (which would mean removing the line of code directly ...
- 230
0votes
2answers
28views
String search on dataframe using key/value from dict
I am trying to match the string present in the below dataframe's 'Disease' column with the key from the dict and if the string is present then change the value in the 'category' column to the value of ...
- 1,122
0votes
1answer
19views
How to use read_excel inside a loop?
when I use read_excel outside a loop it creates a normal dataframe:
pd.read_excel('file.xlsx')
but I need to parse files in order to create dataframes, like:
os.chdir('/path/to/dir')
for file in glob....
- 57
0votes
1answer
17views
In dataframes created by a loop, add suffix to columns that increases with iteration
I am creating multiple dataframes numbered from 1 to n through a loop.
The first thing I do for this, is create a dictionary:
dict_of_df = {}
then runs the loop:
for i in range(1, n+1):
... ...
- 702
0votes
0answers
31views
Python Pandas : How to read huge number of csv files from S3 bucket efficiently
I have 750,000 csv input files in S3 bucket which needs to be transformed and loaded into output folder which will also be stored in csv.
While I have a notebook which reads these many files , while ...
- 11
0votes
4answers
55views
Dataframe: create pairwise difference
I have this dataframe:
Name
Score 1
Score 2
Score 3
...
Score 20
Max
2
3
5
...
2
Jacob
1
4
3
...
6
How do I get the pairwise difference? So the result should look like:
Name
Score 1-2
Score 1-3
...
0votes
0answers
28views
How to append dictionaries to a list in Python?
I have create a dictionary named item_dict.
After iterating the rows of a data frame, where multiple rows have the same id value, I want to append to item_dict a list ,and to that list to add more ...
- 2,320
0votes
1answer
28views
pandas aggregate column doesnt exist?
Currently i have a dataframe that i am preforming a group by on with aggregate functions. these are the functions
aggregation_functions = {
'12_months': 'sum',
'...
- 29
0votes
0answers
15views
Selecting data with boolean
I have this data:
Time Filename 60Ni 61Ni 62Ni 63Cu 64Ni 65Cu 66Zn
0 input/25.03.2022/220310001_Blk.TXT 0.004213561117649 0.0004941140553758 0.0008264054505464 0....
- 25
0votes
0answers
11views
is there a method to verify existence of column in another pandas df? [duplicate]
i have a dataframe like :
Name; address ; firstname ; lastname ; postal_city ; number
Martin Lustig; Paris France; Martin; Fabien ; Paris ; 0
Adrian Dubois; Lille France; Adrian; Dubois; Nantes ; 0
...
0votes
1answer
28views
MASK csv columns using python3
I have below requirement to MASK 3 columns (MAILID,SSN,CCNO) with X (each char with X) using python3, I am trying with lambda function, but not getting exact output. any help on this?
ID NAME ...
- 453
1vote
1answer
22views
if condition not meet, leave blank python code
how should we write the code that tell python to leave empty cell in dataframe when the condition is not meet?
I tries " " like excel but it does not work. I tried 'space' also not work ...
0votes
0answers
13views
Make dataframe with user input and even if user directly press "Enter" without input consider no condition check for particulr column
I'm trying to make a data frame considering user input if user press "Enter" without entering anything. It should not check for condition in particular column
as well user can input one or ...
- 11
0votes
0answers
27views
Pandas cross merge while combining a number of columns (Python)
I am working in python's pandas with 4 dataframes, each of which have a column of unique identifiers followed by multiple columns of shared attributes. I'd like to generate the cartesian product of ...
0votes
2answers
21views
Python Pandas adding a column where the value is across other rows
I would like to use multiple rows in the evaluation of a new column on a pandas dataframe. For a little background lets say that I have rows where data has been aggregated and I need to now how my ...
1vote
1answer
28views
Assign count to unique values in in another column by group values in another column [duplicate]
I have a dataframe that is like this
Group | People
--------------
1 Cindy
1 Dylan
2 Kathy
3 Steven
3 Jonathan
3 Tiffany
And I want to add a new column that adds ...
- 387
0votes
1answer
27views
BMI calculation from two columns of a pandas data frame with missing values
I am still at the beginning of my Python career and I am trying to add a column with the BMI in a DataFrame, which is calculated from two other columns. However, this does not work with my code yet ...
- 51
0votes
0answers
18views
Seaborn heatmap from dataframe with color bars representing metadata
I have the following pandas dataframes:
DF =
USA Canada Denmark Japan England Spain Brazil
mountain 3 1 9 7 1 1 ...
- 25
1vote
1answer
12views
Filtering on product frequency and category
I think there is a fairly simple solution to this but I just can't wrap my head around it right now.
I have a data frame with several hundred thousand orders. I am trying to find the nsmallest margin ...
- 25
0votes
3answers
38views
Using column as tiebreaker for maximums in Python
Reposted with clarification.
I am working on a dataframe that looks like the following:
+-------+----+------+------+
| Value | ID | Date | ID 2 |
+-------+----+------+------+
| 1 | 5 | 2012 | 111 ...
- 71
-2votes
0answers
25views
Convert SAS code to Python showing error in a loop
I am trying to migrate a SAS code to python.
my SAS code is given below:
%do i =1 %to 8;
IF index(_muln_string,"&i.")>1 then _muln_cnt=sum(_muln_cnt,1);
%end;
...
- 139
1vote
2answers
37views
Change field in row if other row has same value in different field
I'm looking for a Python-based solution to a common task I often run into. (I know, there must be cool SQL-magic for this, but... not now.)
Given a table, we want to change the field f2 if oid is the ...
- 151
2votes
1answer
21views
Pandas - Spreading different values in a column on many columns
I have the following table:
Option 1
Option 2
Option 3
A
X
1
B
X
1
A
Y
1
C
X
1
B
Y
1
I need to split the values of each option on different columns based on the translation as follows:
Option 1 ...
- 21
1vote
2answers
33views
How transform list of strings in column and split dataframe by same string to have several?
I have a dataframe with a column containing list of strings.
id sentence category
0 "I love basketball and dunk to the basket" ['...
- 95
0votes
0answers
24views
Python list, last element overwrites entire list
I am working on a program that reads logfiles in, in a weird format.
I have the following code reading from a number of logfiles
(logFiles)
It works fine to itterate through them and so on. The ...
0votes
1answer
19views
Dropping rows from a df rises a SettingWithCopyWarning error (Pandas, Python) when value doesn't exist (using inplace=T)?
I am trying to drop some rows based on a specific value of a dataframe:
dd = {'ae': pd.DataFrame(dict(a=[1,2,4], b=[4,5,6])),
'be': pd.DataFrame(dict(a=[13,21,413], b=[456,54,62]))}
def ...
- 2,970
0votes
1answer
16views
Check if dictionaries are equal in df
I have a df, in which a column contains dictionaries:
a b c d
0 a1 b1 c1 {0.0: 'a', 1.0: 'b'}
1 a2 b2 c2 NaN
2 a3 b3 c3 {0.0: 'cs', 1.0: 'ef', 2.0: 'efg'}
and ...
- 67
0votes
3answers
23views
multi-dimensional dictionary to dataframe python
dict_abc = {'A': [[1, 2, 3], [4, 5, 6], [7, 8, 9]],
'B': [[4, 4, 4], [2, 2, 3],],
'C': [[4, 6, 0]]
}
I would like to convert this to a dataframe in the form
| x | y | z |
...
- 13
0votes
2answers
31views
Remove outliers
I'm working on a housing dataset. Below is my CSV file format:
BHK Location Price
1 A 10
1 A 100
2 B 50
3 C 80
4 A 100
1 C 500
In ...
- 37
0votes
0answers
22views
Changing Data Types of Columns in Pandas Dataframe
This is how the df looks:
User-ID ISBN Book-Rating Book-Title Book-Author
1 "276725" "034545104X" "0"\\n NaN NaN
2 "276726&...
- 1
0votes
0answers
11views
Want to Know alternate Solution of excel2img for linux in python
As we Know excel2img is used to convert excel sheet to image in Windows Only. But i want to convert excel to image on linux server. I already use asposecells (using JVM) but i got half sheet image(i.e ...
0votes
0answers
9views
Is there a way to export multiple pandas Dataframes in different sheet names using "to_csv" [duplicate]
is there a way to export different Dataframes in one single csv "prediction_dataset.csv" file , but in different sheets .
df1 --> sheet_1
df2 --> sheet_2
..
..
dfn --> sheet_n
- 27
-1votes
2answers
31views
How to create two dataframes from a given dataframe?
Assume I have the following data frame:
I want to create two data frames such that for any row if column Actual is equal to column Predicted then the value in both columns goes in one data frame ...
- 9
0votes
0answers
21views
Segmentation fault while using pandas.read_csv
I have a general question regarding what causes Segmentation fault while trying to read csv using pandas.read_csv(). Right now we are trying to analyse the issue but we do not have visibility of the ...
0votes
1answer
16views
How to create a dataframe from extracted hashtags?
I have used below code to extract hashtags from tweets.
def find_tags(row_string):
tags = [x for x in row_string if x.startswith('#')]
return tags
df['split'] = df['text'].str.split(' ')
...
- 461
2votes
3answers
36views
Replacing the first occurrence of a value in each row of a pandas DataFrame
I have a DataFrame like this:
col1
col2
col3
col4
5
7
12
9
0
9
9
1
9
9
1
1
10
5
2
9
9
3
0
18
Each row has at least one 9 and for each row, I want to replace the first instance of it with 90.
...
- 26.4k
0votes
1answer
25views
Remove rows with invalid date and time data [duplicate]
below is my sample dataframe.
I need to remove the columns that have an invalid date or time. Therefore rows 3,5,6 should be deleted. I need a way to verify if the date & time columns contain ...
- 89
0votes
2answers
25views
Parsing dates and times from a large string into seperate columns
I am trying to parse the date and time from a string column.
This is the original column (all one column):
description
4/18/2020 21:05 XXXXXXXXXXXXXXXXXXXXXXXXXXX YYYYYYYYYYYY ZZZZZZZZZZZZZZZZZ
my ...
- 89
0votes
0answers
17views
parallelizing a Python function?
I have a function that submits a search job to a REST API, waits for the API to respond, then downloads 2 sets of JSON data, converts the both JSON's into Pandas dataframes, and returns both ...
- 567
2votes
1answer
31views
Pandas: pivot comma delimited column into multiple columns
I have the following Pandas DataFrame:
import pandas as pd
import numpy as np
df = pd.DataFrame({'id': [1, 2, 3, 4], 'type': ['a,b,c,d', 'b,d', 'c,e', np.nan]})
I need to split the type column based ...
- 57
0votes
1answer
29views
How to get the column name of a dataframe from values in a numpy array
I have a df with 15 columns:
df.columns:
0 class
1 name
2 location
3 income
4 edu_level
--
14 marital_status
after some transformations I got an numpy.ndarray with ...
- 327
1vote
2answers
56views
Calculation on percentage of a percentage
I am trying to calculate percent of percent. per, per1, and per1-per are in percentages
date per per1 per1-per
1/1/2021 0 0 0
2/1/2021 0 0 0
3/1/2021 2 3 ...
- 134
0votes
1answer
35views
Correlation matrix improving print view removing duplicates
when trying to improve the print view of correlation matrix
def view_corr(df):
df = df.unstack()
corr_f = df.sort_values(kind="quicksort", ascending=False)
corr_f = corr_f.dropna(...
- 405