Questions tagged [parallel-processing]

Parallel processing is, in sharp contrast to just a Concurrent processing, guaranteed to start / perform / finish all thread-level and/or instruction-level tasks executed in a parallel fashion and provides a guaranteed finish of the simultaneously executed code-paths.

Filter by
Sorted by
Tagged with
1vote
0answers
27views

double free or corruption (fasttop) when using OpenMP for multi-threaded programming

The following function in my code gives me this message: *** Error in `./test_exe': double free or corruption (fasttop): 0x0000000002050510 *** The program ran well if I only use 1 thread, but it got ...
user avatar
  • 564
0votes
0answers
9views

How to parallelize merge sort using JavaRDD<Integer> Spark?

public class MergeSort { public static List<Integer> serialSort(List<Integer> array) { if (array.size() > 1) { int mid = array.size() / 2; List<Integer> ...
user avatar
  • 1
0votes
0answers
7views

want to connect to mysql from parallel desktop

I wanna connect to mysql database which is on my Mac from parallel desktop(windows 11) ,but I can't. How can I connect to Mac IP from my parallel? (I tried to install mysql workbench on parallel but ...
user avatar
0votes
0answers
11views

Programmatically setting number of processes with ray

I want to use Ray to parallelize some computations in python. As part of this, I want a method which takes the desired number of worker processes as an argument. The introductory articles on Ray that ...
user avatar
-1votes
0answers
13views

How numpy limits the number of CPU cores

My CPU is multi-core. I want numpy to execute on a single core or with a single thread. What should I do? The configuration is as follows. numpy.show_config() blas_mkl_info: NOT AVAILABLE blis_info: ...
user avatar
  • 1
0votes
0answers
35views

Best way to use parallel computing within a for-loop in Python

I am new to parallel computing and am trying to speed up my code. I have read about several possibilities, using Dask or Tensorflow for example, but I struggle to see what the most efficient method is ...
user avatar
0votes
0answers
38views

Why my allreduce using ring alforithm cannot work?

I am implementing mpi function in allreduce in mpi using ring algorithm. I write function Ring_Allreduce and Naive_Allreduce is a naive implement and also use MPI_Allreduce for test. And below is my ...
user avatar
0votes
0answers
30views

Oralce uncommitted data in other sessions violating unique constraint leading to never-ending query

I am running into never ending update when there's uncommitted data in other session violating unique constraint. Here's the scenario: Say a table t has a unique contraint on one of its columns: ...
user avatar
1vote
1answer
31views

Parallelize a bash script and wait for each loop to finish

I'm trying to write a script, that we call pippo.R. pippo.R aim, is to run another script (for.sh) in a for loop with a parallelization using two values : nPerm= total number of times the script has ...
user avatar
0votes
1answer
28views

How to fork and merge a REST API call that takes a list of id's as query param?

I have to call a third party GET API which takes a comma seperated list of ids and returns a JSON response. Like so: def call(sysid) # RestClient GET https://test.com?ids=sysid.join(',')&.... ...
user avatar
  • 1,697
0votes
0answers
18views

Oracle pipe lined functions

We are using pipelined function to return the results of a select statement, pipe row will return the output of select statement. The time taken to execute each select statement that is returned by ...
user avatar
0votes
0answers
19views

Distributed / parallel streaming / piping to memory-mapped / external sort with julia (emulating GNU Parallel / bash for any OS)

What would be the best way to implement the following bash, in Julia? zstd -dc $INPUTFILE \ | parallel -j $NPROCS --pipe python process.py \ | sort -T /dev/shm/ --parallel=8 -k2,2 \ | pzstd > ...
user avatar
  • 3,281
0votes
0answers
17views

parallelizing a Python function?

I have a function that submits a search job to a REST API, waits for the API to respond, then downloads 2 sets of JSON data, converts the both JSON's into Pandas dataframes, and returns both ...
user avatar
0votes
0answers
8views

Nginx upstream routing based on value of cookie

This one is driving me bonkers as it is feeling like variables are not being used during the final routing, I must be missing something simple. The goal: The viewer will have a cookie called ...
user avatar
0votes
2answers
76views

Python parallelized code is taking much longer than it should

EDIT: My goal with this post is to understand the source of the time-drain. I welcome other recommendations as well, but my main concern, and I want to learn about is why my code is not speeding up ...
user avatar
0votes
0answers
24views

R Error:cons memory exhausted ~530GB of 1.35TB avail RAM, why?

After many attempts, I still cannot solve this error. First, here are my specs: R version 4.1.3 (2022-03-10) -- "One Push-Up" Platform: x86_64-pc-linux-gnu (64-bit) Linux 5.13.0-37-generic #...
user avatar
  • 5
0votes
2answers
54views

Efficiency of using omp parallel for inside omp parallel omp task

I am trying to improve the efficiency of a complicated project using OpenMP. I will use a toy example to show the situation I am facing. First, I have a recursion function and I use omp task to ...
user avatar
  • 564
-2votes
0answers
19views

Syncronize access to dynamodb from parallel threads

I'm working with FastAPI to develop a python API over AWS. In one of my api calls it takes long to return values (10s). Therefore, I removed async when I defined my function api so they will run in ...
user avatar
0votes
0answers
20views

PowerShell 7's Parallel ForEach-Object + Pester's mocking

I'm trying to write a regression test for an issue I have with Invoke-WebRequest accessing the same file, while running in parallel (separate PS sessions). Invoke-WebRequest call is part of a larger ...
user avatar
  • 185
1vote
0answers
24views

TransactionScope() and parallel query execution

We're trying to run parallel queries inside a transaction scope to improve performance of our code. We have several changes to be made in the database that have no connection to each other. We could ...
user avatar
  • 1,417
-1votes
2answers
45views

How do I implement this algorithm in C++ using OpenMP, or in which application can I implement it?

How do I implement Sum_EREW algorithm in C++ using OpenMP, or in which application can I implement it? for i = 1 to log (n) do forall j where 1 <= j <= n/2 do in parallel if (2j ...
user avatar
-1votes
0answers
21views

I am trying to do matrix multiplication using pthreads and I am trying to calculate the time taken for different threads

#include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <pthread.h> #include <time.h> #define N 3000 // number of rows of matrix A #define M 1 // number of ...
user avatar
1vote
3answers
69views

Python: _pickle.PicklingError: Can't pickle <function <lambda>>

I'm running Python 3.9.1 Note: I know there are questions with similar titles. But those questions are embedded in complicated code that makes it hard to understand the problem. This is a bare-bones ...
user avatar
1vote
0answers
14views

Is it possible to parallelize the geocode function from the tidygeocoder package in R?

I am trying to geocode a couple of addresses in R. I am working with the geocode function from the tidygeocoder package in R. To make it faster I'd love to parallelize this function, but I do not know ...
user avatar
0votes
1answer
7views

Xpress Mosel Parallel Solving

Is it possible to have two submodles (.mos) running in parallel as input data for a master problem? Is it correct to repeat twice the next code, one for each sub model? Because I have one bim file for ...
user avatar
0votes
1answer
54views

Spawning Isolates in for loop and processing parallelly in Dart

for(int i=0;i<2;i++){ ..............preprocessing(pr)........... ..... = await someFunc(someData[i]) (this line spawns off new Isolate) ..............postprocessing(po).......... } I ...
user avatar
1vote
0answers
10views

How to solve R future error "error in future$uuid : object of type closure is not subsettable"

When trying to register a future with a parallel backend in R, I receive the following error message: Error in future$uuid : object of type 'closure' is not subsettable Previously using future has ...
user avatar
1vote
1answer
31views

I am using pthreads to speed up my matrix multiplication but not getting correct values

I have to calculate the speed of operation for different threads but the matrix size has to be (1000X1 and 1X1000). I have to calculate this using 1,2,4,8,16,32,64,128,256 and 512 threads. my program ...
user avatar
-1votes
0answers
24views

How can I listen on serial port and use GPIO at the same time

I want to use a light sensor on the GPIO port of rPI, and want to get some incoming data on the console port at the same time. Depending on the values that I got on the console may I store or not the ...
user avatar
0votes
1answer
15views

Submitting parallel jobs on HTCondor, using python

I am trying to submit parallel jobs in a loop on HTCondor, following is a simple example of the python script - test_mus = np.linspace(0, 5, 10) results = [pyhf.infer.hypotest(test_mu, data, model) ...
user avatar
  • 13
0votes
1answer
44views

What is the best way to parallelise this code?

I have a function that I would like to maximise using python. However, the evaluation of this function is fairly slow and I would like to find a way to speed it up by parallelising this code. I'm not ...
user avatar
  • 73
0votes
0answers
10views

R parallel: Using `source` output as input in a function within the main function

Apologies in advance, I will use pseudo-code. I tried to create a reprex but it didn't show the problem. I am using the function parallel::parSapply to run a model and it's throwing an "object ...
user avatar
2votes
2answers
38views

Parallel map with nested functions that depend on modules

Here's a minimal example of what I'm trying to parallelize import numpy as np def generate_function(a): def func(x): '''a complex function that uses several modules''' return x + ...
user avatar
  • 682
0votes
0answers
29views

Openmp's unexpected unspecified behavior when testing values after parallel block

Consider the following: // compile with // gcc -fopenmp test.c #include <stdio.h> // to display on the screen #include <stdlib.h> // to access EXIT_SUCCESS #include <omp.h> ...
user avatar
  • 1,708
3votes
1answer
39views

Difference between "parallelStream()" and "asSequence().asStream().parallel()"

Using kotlin 1.6.10 it seems that parallelStream() and asSequence().asStream().parallel() are not behaving the same. The latter doesn't seem to parallelize. Here is an example code snippet, ...
user avatar
  • 1,464
0votes
1answer
23views

openpyxl performance in read-only mode

I have a question about the performance of openpyxl when reading files. I am trying to read the same xlsx file using ProcessPoolExecutor, single file Maybe 500,000 to 800,000 rows. In read-only mode ...
user avatar
0votes
0answers
25views

AMP C++ index<1> iteration

I have written other code which outputs different accelerator info, however I am having a bit of trouble understanding completely the following below. I am having a bit of a hard time understanding ...
user avatar
0votes
1answer
15views

Jmeter - Websocket plugin - unable to find solution to constantly read incoming messages

Good afternoon. I have implemented websockets in my project using the plugin "WebSocket Samplers by Peter Doornbosch" in Jmeter. So far the requirements were simply to send and receive some ...
user avatar
  • 1
0votes
1answer
32views

multiprocessing with 5 processes slower than single process on an old 2-core CPU

I am trying to run 5 parallel processes and have also divided the list in 5 blocks but it takes more time than linearly processing in a single process, I have tried other solutions on Stackoverflow ...
user avatar
0votes
1answer
28views

How to kill all parallel Bash/Linux tasks on condition of single instance

I have a parallelised Bash function: #!/bin/bash touch out.txt vals=$(seq 1 1 20) task(){ echo $1 >> out.txt x=$(wc -l out.txt | cut -d' ' -f 1) #read current length of file if [ $x ...
user avatar
1vote
0answers
33views

Parallelization of loop with multiple arguments python

I have a loop that takes minimum 2 hours to run, so I decided to parallelize it. I never paralleled with python but heard multiprocessing is reliable therefore I tried it but when I try to use starmap ...
user avatar
  • 84
0votes
1answer
45views

Use thread-specific pre-allocated data to run tasks in parallel

I want to run some tasks in parallel. Each task uses heap-allocated data. To speed things up, I would like that each thread re-use the same data instead of un-allocating it and re-allocating it just ...
user avatar
0votes
0answers
22views

While loop inside function given to multiprocessing pool not working

I am trying to implement a simple api call, in parallel, but for some reason, it is calling all of the data that I provided in parallel, instead of the number I defined in Pool, i.e, Pool(6). I am ...
user avatar
-1votes
1answer
38views

Python parallel processing fills memory fast

I'm bruteforcing a 8-digit pin on a ELF executable (it's for a CTF) and I'm using asynchronous parallel processing. The code is very fast but it fills the memory even faster. It takes about 10% of the ...
user avatar
-1votes
0answers
29views

Cluster computing with gpus and cpus in each node

I have a python program which needs tons of power, as it needs to loop around 900 to 3,000 times per second depending on load, and to do so I need a ton of power, because the program isn't light. It ...
user avatar
-1votes
0answers
11views

how to parallel the pure-ftpd CallUploadScript?

I'm using pure-ftpd version 1.0.47 for server and a lftp version 4.4.8 for client (i have 9 clients) in order to upload files to the server. The average amount of files per second is 100 with a size ...
user avatar
1vote
2answers
67views

Parallel.ForEach with Priority Queue in .NET6

I'm trying to run Parallel.ForEach on my Priority Queue but I am getting the following error: Severity Code Description Project File Line Suppression State Error CS0411 The type ...
user avatar
0votes
2answers
43views

Intel MKL ERROR: incorrect parameter when calling gemm()

I have this code: void my_function(double *image_vector, double *endmembers, double *abundanceVector, int it, int lines, int samples, int bands, int targets) { double *h_Num; double *h_aux; ...
user avatar
2votes
1answer
42views

Powershell 7 -> ForEach -Parallel in a Function does not return anything when the result is added to the array

I need to use Powershell 7 Parallel looping feature in this function but when using ForEach loop, I cannot take the result and put it into the array at the end and I do not undestand why. Any ideas? ...
user avatar
  • 189
0votes
1answer
61views

Is there an efficient algorithm to redistribute a vector over processes based on how many elements each process has to give up or recieve?

I'm trying to redistribute an array (mesh-like) over a set of processes for load-balancing needs. My special requirement is that array elements should only be moved to the spatially adjacent processes ...
user avatar
  • 71

15 30 50 per page
1
2 3 4 5
395